2024-01-18T06:34:58.985 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-18T06:34:58.985 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2024-01-17_22:43:41-rados-wip-yuri10-testing-2024-01-17-0759-pacific-distro-default-smithi/7519629 --verbose --name yuriw-2024-01-17_22:43:41-rados-wip-yuri10-testing-2024-01-17-0759-pacific-distro-default-smithi --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2024-01-17_22:43:41-rados-wip-yuri10-testing-2024-01-17-0759-pacific-distro-default-smithi/7519629/orig.config.yaml 2024-01-18T06:34:59.035 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-18T06:34:59.125 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-01-17_22:43:41-rados-wip-yuri10-testing-2024-01-17-0759-pacific-distro-default-smithi/7519629 branch: wip-yuri10-testing-2024-01-17-0759-pacific description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: yweinste@redhat.com first_in_suite: false job_id: '7519629' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2024-01-17_22:43:41-rados-wip-yuri10-testing-2024-01-17-0759-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri10-testing-2024-01-17-0759-pacific ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri10-testing-2024-01-17-0759-pacific sha1: 491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 2168 sha1: 491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri10-testing-2024-01-17-0759-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0 targets: smithi047.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDwqTlovo+UQwwudhGXWkdlbOj3jbu+mFjgFuy+Gum24E77mpPYOVbPuFmy3dpfKHHDBKtVCTwbC5ZH+n8ZXucI= smithi194.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO6tf9GNGxt7c16eAsvnjeb9dAGmQe1i8V7sc2WV7fuPyEFm1sh0M3FFdYCRyQvEp0ntsTICFFtBzvDczjuRGD4= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: cd45576300487d997e5a85abed65500b9f5d143b timestamp: 2024-01-17_22:43:41 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-01-18T06:34:59.125 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0/qa; will attempt to use it 2024-01-18T06:34:59.126 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0/qa/tasks 2024-01-18T06:34:59.126 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-18T06:34:59.128 INFO:teuthology.task.internal:Checking packages... 2024-01-18T06:34:59.149 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0' 2024-01-18T06:34:59.149 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-18T06:34:59.149 INFO:teuthology.packaging:ref: None 2024-01-18T06:34:59.149 INFO:teuthology.packaging:tag: None 2024-01-18T06:34:59.149 INFO:teuthology.packaging:branch: wip-yuri10-testing-2024-01-17-0759-pacific 2024-01-18T06:34:59.149 INFO:teuthology.packaging:sha1: 491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0 2024-01-18T06:34:59.149 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri10-testing-2024-01-17-0759-pacific 2024-01-18T06:34:59.297 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-454.g491b5e13 2024-01-18T06:34:59.299 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-18T06:34:59.306 INFO:teuthology.task.internal:no buildpackages task found 2024-01-18T06:34:59.306 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-18T06:34:59.329 INFO:teuthology.task.internal:Saving configuration 2024-01-18T06:34:59.348 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-18T06:34:59.356 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-18T06:34:59.382 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi047.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-17_22:43:41-rados-wip-yuri10-testing-2024-01-17-0759-pacific-distro-default-smithi/7519629', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-18 06:29:59.808262', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDwqTlovo+UQwwudhGXWkdlbOj3jbu+mFjgFuy+Gum24E77mpPYOVbPuFmy3dpfKHHDBKtVCTwbC5ZH+n8ZXucI='} 2024-01-18T06:34:59.406 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi194.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-17_22:43:41-rados-wip-yuri10-testing-2024-01-17-0759-pacific-distro-default-smithi/7519629', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-18 06:29:59.806539', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO6tf9GNGxt7c16eAsvnjeb9dAGmQe1i8V7sc2WV7fuPyEFm1sh0M3FFdYCRyQvEp0ntsTICFFtBzvDczjuRGD4='} 2024-01-18T06:34:59.406 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-18T06:34:59.412 INFO:teuthology.task.internal:roles: ubuntu@smithi047.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-01-18T06:34:59.412 INFO:teuthology.task.internal:roles: ubuntu@smithi194.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-01-18T06:34:59.412 INFO:teuthology.run_tasks:Running task console_log... 2024-01-18T06:34:59.483 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f54cca57ee0>, signals=[15]) 2024-01-18T06:34:59.483 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-18T06:34:59.491 INFO:teuthology.task.internal:Opening connections... 2024-01-18T06:34:59.491 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi047.front.sepia.ceph.com 2024-01-18T06:34:59.493 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-18T06:34:59.565 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi194.front.sepia.ceph.com 2024-01-18T06:34:59.566 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-18T06:34:59.635 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-18T06:34:59.644 DEBUG:teuthology.orchestra.run.smithi047:> uname -m 2024-01-18T06:34:59.667 INFO:teuthology.orchestra.run.smithi047.stdout:x86_64 2024-01-18T06:34:59.667 DEBUG:teuthology.orchestra.run.smithi047:> cat /etc/os-release 2024-01-18T06:34:59.728 INFO:teuthology.orchestra.run.smithi047.stdout:NAME="CentOS Stream" 2024-01-18T06:34:59.728 INFO:teuthology.orchestra.run.smithi047.stdout:VERSION="8" 2024-01-18T06:34:59.729 INFO:teuthology.orchestra.run.smithi047.stdout:ID="centos" 2024-01-18T06:34:59.729 INFO:teuthology.orchestra.run.smithi047.stdout:ID_LIKE="rhel fedora" 2024-01-18T06:34:59.729 INFO:teuthology.orchestra.run.smithi047.stdout:VERSION_ID="8" 2024-01-18T06:34:59.729 INFO:teuthology.orchestra.run.smithi047.stdout:PLATFORM_ID="platform:el8" 2024-01-18T06:34:59.729 INFO:teuthology.orchestra.run.smithi047.stdout:PRETTY_NAME="CentOS Stream 8" 2024-01-18T06:34:59.729 INFO:teuthology.orchestra.run.smithi047.stdout:ANSI_COLOR="0;31" 2024-01-18T06:34:59.729 INFO:teuthology.orchestra.run.smithi047.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-01-18T06:34:59.729 INFO:teuthology.orchestra.run.smithi047.stdout:HOME_URL="https://centos.org/" 2024-01-18T06:34:59.729 INFO:teuthology.orchestra.run.smithi047.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-18T06:34:59.729 INFO:teuthology.orchestra.run.smithi047.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-18T06:34:59.729 INFO:teuthology.orchestra.run.smithi047.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-18T06:34:59.730 INFO:teuthology.lock.ops:Updating smithi047.front.sepia.ceph.com on lock server 2024-01-18T06:34:59.755 DEBUG:teuthology.orchestra.run.smithi194:> uname -m 2024-01-18T06:34:59.777 INFO:teuthology.orchestra.run.smithi194.stdout:x86_64 2024-01-18T06:34:59.778 DEBUG:teuthology.orchestra.run.smithi194:> cat /etc/os-release 2024-01-18T06:34:59.839 INFO:teuthology.orchestra.run.smithi194.stdout:NAME="CentOS Stream" 2024-01-18T06:34:59.839 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION="8" 2024-01-18T06:34:59.839 INFO:teuthology.orchestra.run.smithi194.stdout:ID="centos" 2024-01-18T06:34:59.839 INFO:teuthology.orchestra.run.smithi194.stdout:ID_LIKE="rhel fedora" 2024-01-18T06:34:59.839 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION_ID="8" 2024-01-18T06:34:59.839 INFO:teuthology.orchestra.run.smithi194.stdout:PLATFORM_ID="platform:el8" 2024-01-18T06:34:59.839 INFO:teuthology.orchestra.run.smithi194.stdout:PRETTY_NAME="CentOS Stream 8" 2024-01-18T06:34:59.839 INFO:teuthology.orchestra.run.smithi194.stdout:ANSI_COLOR="0;31" 2024-01-18T06:34:59.839 INFO:teuthology.orchestra.run.smithi194.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-01-18T06:34:59.839 INFO:teuthology.orchestra.run.smithi194.stdout:HOME_URL="https://centos.org/" 2024-01-18T06:34:59.839 INFO:teuthology.orchestra.run.smithi194.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-18T06:34:59.840 INFO:teuthology.orchestra.run.smithi194.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-18T06:34:59.840 INFO:teuthology.orchestra.run.smithi194.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-18T06:34:59.840 INFO:teuthology.lock.ops:Updating smithi194.front.sepia.ceph.com on lock server 2024-01-18T06:34:59.863 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-18T06:34:59.873 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-18T06:34:59.881 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-18T06:34:59.881 DEBUG:teuthology.orchestra.run.smithi047:> test '!' -e /home/ubuntu/cephtest 2024-01-18T06:34:59.883 DEBUG:teuthology.orchestra.run.smithi194:> test '!' -e /home/ubuntu/cephtest 2024-01-18T06:34:59.904 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-18T06:34:59.912 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-18T06:34:59.912 DEBUG:teuthology.orchestra.run.smithi047:> test -z $(ls -A /var/lib/ceph) 2024-01-18T06:34:59.944 DEBUG:teuthology.orchestra.run.smithi194:> test -z $(ls -A /var/lib/ceph) 2024-01-18T06:34:59.973 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-18T06:35:00.030 INFO:teuthology.run_tasks:Running task kernel... 2024-01-18T06:35:00.045 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-01-18T06:35:00.045 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-01-18T06:35:00.046 DEBUG:teuthology.orchestra.run.smithi047:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-18T06:35:00.046 DEBUG:teuthology.orchestra.run.smithi194:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-18T06:35:00.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-18T06:35:00.066 DEBUG:teuthology.orchestra.run.smithi194:> uname -r 2024-01-18T06:35:00.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-18T06:35:00.067 DEBUG:teuthology.orchestra.run.smithi047:> uname -r 2024-01-18T06:35:00.127 INFO:teuthology.orchestra.run.smithi194.stdout:4.18.0-489.el8.x86_64 2024-01-18T06:35:00.127 INFO:teuthology.task.kernel:Running kernel on smithi194: 4.18.0-489.el8.x86_64 2024-01-18T06:35:00.127 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum install -y kernel 2024-01-18T06:35:00.130 INFO:teuthology.orchestra.run.smithi047.stdout:4.18.0-489.el8.x86_64 2024-01-18T06:35:00.130 INFO:teuthology.task.kernel:Running kernel on smithi047: 4.18.0-489.el8.x86_64 2024-01-18T06:35:00.130 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum install -y kernel 2024-01-18T06:35:03.448 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 8 - AppStream 31 MB/s | 26 MB 00:00 2024-01-18T06:35:03.693 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 8 - AppStream 32 MB/s | 26 MB 00:00 2024-01-18T06:35:08.956 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 8 - BaseOS 12 MB/s | 10 MB 00:00 2024-01-18T06:35:09.566 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 8 - BaseOS 11 MB/s | 10 MB 00:00 2024-01-18T06:35:11.081 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 8 - Extras 33 kB/s | 18 kB 00:00 2024-01-18T06:35:11.883 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 8 - Extras 29 kB/s | 18 kB 00:00 2024-01-18T06:35:14.506 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 8 - Extras common packages 2.1 kB/s | 6.9 kB 00:03 2024-01-18T06:35:14.600 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 8 - Extras common packages 2.7 kB/s | 6.9 kB 00:02 2024-01-18T06:35:14.866 INFO:teuthology.orchestra.run.smithi194.stdout:Copr repo for python3-asyncssh owned by ceph 13 kB/s | 3.5 kB 00:00 2024-01-18T06:35:14.969 INFO:teuthology.orchestra.run.smithi047.stdout:Copr repo for python3-asyncssh owned by ceph 13 kB/s | 3.5 kB 00:00 2024-01-18T06:35:15.719 INFO:teuthology.orchestra.run.smithi194.stdout:Extra Packages for Enterprise Linux 21 MB/s | 16 MB 00:00 2024-01-18T06:35:18.565 INFO:teuthology.orchestra.run.smithi047.stdout:Extra Packages for Enterprise Linux 4.6 MB/s | 16 MB 00:03 2024-01-18T06:35:19.483 INFO:teuthology.orchestra.run.smithi194.stdout:lab-extras 467 kB/s | 24 kB 00:00 2024-01-18T06:35:22.380 INFO:teuthology.orchestra.run.smithi194.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-01-18T06:35:22.380 INFO:teuthology.orchestra.run.smithi194.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-01-18T06:35:22.478 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-18T06:35:22.478 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:35:22.479 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-01-18T06:35:22.479 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:35:22.479 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-01-18T06:35:22.479 INFO:teuthology.orchestra.run.smithi194.stdout: kernel x86_64 4.18.0-535.el8 baseos 10 M 2024-01-18T06:35:22.479 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-01-18T06:35:22.479 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-core x86_64 4.18.0-535.el8 baseos 43 M 2024-01-18T06:35:22.479 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-modules x86_64 4.18.0-535.el8 baseos 36 M 2024-01-18T06:35:22.479 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:35:22.479 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-01-18T06:35:22.479 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:35:22.479 INFO:teuthology.orchestra.run.smithi194.stdout:Install 3 Packages 2024-01-18T06:35:22.479 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:35:22.480 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 89 M 2024-01-18T06:35:22.480 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 96 M 2024-01-18T06:35:22.480 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-01-18T06:35:22.510 INFO:teuthology.orchestra.run.smithi047.stdout:lab-extras 533 kB/s | 24 kB 00:00 2024-01-18T06:35:23.393 INFO:teuthology.orchestra.run.smithi194.stdout:(1/3): kernel-4.18.0-535.el8.x86_64.rpm 12 MB/s | 10 MB 00:00 2024-01-18T06:35:24.502 INFO:teuthology.orchestra.run.smithi194.stdout:(2/3): kernel-modules-4.18.0-535.el8.x86_64.rpm 18 MB/s | 36 MB 00:01 2024-01-18T06:35:25.010 INFO:teuthology.orchestra.run.smithi194.stdout:(3/3): kernel-core-4.18.0-535.el8.x86_64.rpm 17 MB/s | 43 MB 00:02 2024-01-18T06:35:25.010 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-01-18T06:35:25.010 INFO:teuthology.orchestra.run.smithi194.stdout:Total 35 MB/s | 89 MB 00:02 2024-01-18T06:35:25.540 INFO:teuthology.orchestra.run.smithi047.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-01-18T06:35:25.541 INFO:teuthology.orchestra.run.smithi047.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-01-18T06:35:25.657 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-01-18T06:35:25.658 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:35:25.658 INFO:teuthology.orchestra.run.smithi047.stdout: Package Architecture Version Repository Size 2024-01-18T06:35:25.658 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:35:25.658 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-01-18T06:35:25.658 INFO:teuthology.orchestra.run.smithi047.stdout: kernel x86_64 4.18.0-535.el8 baseos 10 M 2024-01-18T06:35:25.658 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-01-18T06:35:25.658 INFO:teuthology.orchestra.run.smithi047.stdout: kernel-core x86_64 4.18.0-535.el8 baseos 43 M 2024-01-18T06:35:25.659 INFO:teuthology.orchestra.run.smithi047.stdout: kernel-modules x86_64 4.18.0-535.el8 baseos 36 M 2024-01-18T06:35:25.659 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:35:25.659 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-01-18T06:35:25.659 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:35:25.659 INFO:teuthology.orchestra.run.smithi047.stdout:Install 3 Packages 2024-01-18T06:35:25.659 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:35:25.659 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 89 M 2024-01-18T06:35:25.659 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 96 M 2024-01-18T06:35:25.659 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-01-18T06:35:25.699 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-01-18T06:35:26.104 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-01-18T06:35:26.104 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-01-18T06:35:27.339 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-01-18T06:35:27.360 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-01-18T06:35:27.695 INFO:teuthology.orchestra.run.smithi047.stdout:(1/3): kernel-4.18.0-535.el8.x86_64.rpm 5.3 MB/s | 10 MB 00:01 2024-01-18T06:35:28.428 INFO:teuthology.orchestra.run.smithi047.stdout:(2/3): kernel-core-4.18.0-535.el8.x86_64.rpm 16 MB/s | 43 MB 00:02 2024-01-18T06:35:29.137 INFO:teuthology.orchestra.run.smithi047.stdout:(3/3): kernel-modules-4.18.0-535.el8.x86_64.rpm 11 MB/s | 36 MB 00:03 2024-01-18T06:35:29.137 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-01-18T06:35:29.137 INFO:teuthology.orchestra.run.smithi047.stdout:Total 26 MB/s | 89 MB 00:03 2024-01-18T06:35:29.849 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-01-18T06:35:30.077 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-01-18T06:35:30.307 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-01-18T06:35:30.308 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-01-18T06:35:30.547 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-core-4.18.0-535.el8.x86_64 1/3 2024-01-18T06:35:31.605 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-01-18T06:35:31.638 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-01-18T06:35:34.147 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-core-4.18.0-535.el8.x86_64 1/3 2024-01-18T06:35:34.521 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-modules-4.18.0-535.el8.x86_64 2/3 2024-01-18T06:35:34.573 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-01-18T06:35:35.110 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : kernel-core-4.18.0-535.el8.x86_64 1/3 2024-01-18T06:35:37.943 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-modules-4.18.0-535.el8.x86_64 2/3 2024-01-18T06:35:38.228 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-4.18.0-535.el8.x86_64 3/3 2024-01-18T06:35:38.906 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: kernel-core-4.18.0-535.el8.x86_64 1/3 2024-01-18T06:35:39.318 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : kernel-modules-4.18.0-535.el8.x86_64 2/3 2024-01-18T06:35:42.993 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: kernel-modules-4.18.0-535.el8.x86_64 2/3 2024-01-18T06:35:43.286 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : kernel-4.18.0-535.el8.x86_64 3/3 2024-01-18T06:36:00.337 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-core-4.18.0-535.el8.x86_64 3/3 2024-01-18T06:36:00.362 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-modules-4.18.0-535.el8.x86_64 3/3 2024-01-18T06:36:00.711 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-4.18.0-535.el8.x86_64 3/3 2024-01-18T06:36:00.711 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-4.18.0-535.el8.x86_64 1/3 2024-01-18T06:36:00.711 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-core-4.18.0-535.el8.x86_64 2/3 2024-01-18T06:36:01.176 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-modules-4.18.0-535.el8.x86_64 3/3 2024-01-18T06:36:01.176 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:36:01.176 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-01-18T06:36:01.176 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-4.18.0-535.el8.x86_64 kernel-core-4.18.0-535.el8.x86_64 2024-01-18T06:36:01.176 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-modules-4.18.0-535.el8.x86_64 2024-01-18T06:36:01.176 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:36:01.176 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-18T06:36:01.313 DEBUG:teuthology.orchestra.run.smithi194:> rpm -q kernel | sort -rV | head -n 1 2024-01-18T06:36:01.567 INFO:teuthology.orchestra.run.smithi194.stdout:kernel-4.18.0-535.el8.x86_64 2024-01-18T06:36:01.568 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-535.el8.x86_64 2024-01-18T06:36:01.568 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-535.el8.x86_64 2024-01-18T06:36:01.568 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-18T06:36:01.568 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-18T06:36:01.569 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-01-18T06:36:01.569 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi194.front.sepia.ceph.com, path=None, version=distro) 2024-01-18T06:36:01.569 DEBUG:teuthology.orchestra.run.smithi194:> rpm -q kernel | sort -rV | head -n 1 2024-01-18T06:36:01.778 INFO:teuthology.orchestra.run.smithi194.stdout:kernel-4.18.0-535.el8.x86_64 2024-01-18T06:36:01.779 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-535.el8.x86_64 2024-01-18T06:36:01.779 DEBUG:teuthology.orchestra.run.smithi194:> sudo rpm -qi grub2-tools 2024-01-18T06:36:01.811 INFO:teuthology.orchestra.run.smithi194.stdout:Name : grub2-tools 2024-01-18T06:36:01.812 INFO:teuthology.orchestra.run.smithi194.stdout:Epoch : 1 2024-01-18T06:36:01.812 INFO:teuthology.orchestra.run.smithi194.stdout:Version : 2.02 2024-01-18T06:36:01.812 INFO:teuthology.orchestra.run.smithi194.stdout:Release : 129.el8 2024-01-18T06:36:01.812 INFO:teuthology.orchestra.run.smithi194.stdout:Architecture: x86_64 2024-01-18T06:36:01.812 INFO:teuthology.orchestra.run.smithi194.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-01-18T06:36:01.812 INFO:teuthology.orchestra.run.smithi194.stdout:Group : System Environment/Base 2024-01-18T06:36:01.812 INFO:teuthology.orchestra.run.smithi194.stdout:Size : 9265586 2024-01-18T06:36:01.812 INFO:teuthology.orchestra.run.smithi194.stdout:License : GPLv3+ 2024-01-18T06:36:01.812 INFO:teuthology.orchestra.run.smithi194.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-01-18T06:36:01.812 INFO:teuthology.orchestra.run.smithi194.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-01-18T06:36:01.813 INFO:teuthology.orchestra.run.smithi194.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-01-18T06:36:01.813 INFO:teuthology.orchestra.run.smithi194.stdout:Build Host : kbuilder.bsys.centos.org 2024-01-18T06:36:01.813 INFO:teuthology.orchestra.run.smithi194.stdout:Relocations : (not relocatable) 2024-01-18T06:36:01.813 INFO:teuthology.orchestra.run.smithi194.stdout:Packager : CentOS BuildSystem 2024-01-18T06:36:01.813 INFO:teuthology.orchestra.run.smithi194.stdout:Vendor : CentOS 2024-01-18T06:36:01.813 INFO:teuthology.orchestra.run.smithi194.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-18T06:36:01.813 INFO:teuthology.orchestra.run.smithi194.stdout:Summary : Support tools for GRUB. 2024-01-18T06:36:01.813 INFO:teuthology.orchestra.run.smithi194.stdout:Description : 2024-01-18T06:36:01.813 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:36:01.813 INFO:teuthology.orchestra.run.smithi194.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-18T06:36:01.813 INFO:teuthology.orchestra.run.smithi194.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-18T06:36:01.813 INFO:teuthology.orchestra.run.smithi194.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-18T06:36:01.813 INFO:teuthology.orchestra.run.smithi194.stdout:hardware devices. 2024-01-18T06:36:01.813 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:36:01.813 INFO:teuthology.orchestra.run.smithi194.stdout:This subpackage provides tools for support of all platforms. 2024-01-18T06:36:01.814 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-18T06:36:01.814 INFO:teuthology.task.kernel:Updating grub on smithi194 to boot 4.18.0-535.el8.x86_64 2024-01-18T06:36:01.815 DEBUG:teuthology.orchestra.run.smithi194:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-18T06:36:03.206 INFO:teuthology.orchestra.run.smithi194.stderr:Generating grub configuration file ... 2024-01-18T06:36:05.674 INFO:teuthology.orchestra.run.smithi194.stderr:done 2024-01-18T06:36:05.676 DEBUG:teuthology.orchestra.run.smithi194:> mktemp 2024-01-18T06:36:05.708 INFO:teuthology.orchestra.run.smithi194.stdout:/tmp/tmp.LkxJZJAypk 2024-01-18T06:36:05.709 DEBUG:teuthology.orchestra.run.smithi194:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.LkxJZJAypk 2024-01-18T06:36:05.782 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 0666 /tmp/tmp.LkxJZJAypk 2024-01-18T06:36:05.927 DEBUG:teuthology.orchestra.remote:smithi194:/tmp/tmp.LkxJZJAypk is 6KB 2024-01-18T06:36:05.976 DEBUG:teuthology.orchestra.run.smithi194:> rm -fr /tmp/tmp.LkxJZJAypk 2024-01-18T06:36:05.993 DEBUG:teuthology.orchestra.run.smithi194:> sudo /bin/ls /boot/loader/entries || true 2024-01-18T06:36:06.059 INFO:teuthology.orchestra.run.smithi194.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-01-18T06:36:06.059 INFO:teuthology.orchestra.run.smithi194.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-01-18T06:36:06.059 INFO:teuthology.orchestra.run.smithi194.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-01-18T06:36:06.059 INFO:teuthology.orchestra.run.smithi194.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-535.el8.x86_64.conf 2024-01-18T06:36:06.060 DEBUG:teuthology.orchestra.run.smithi194:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-535.el8.x86_64 2024-01-18T06:36:06.199 DEBUG:teuthology.orchestra.run.smithi194:> sudo shutdown -r now 2024-01-18T06:36:06.837 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: kernel-core-4.18.0-535.el8.x86_64 3/3 2024-01-18T06:36:06.862 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: kernel-modules-4.18.0-535.el8.x86_64 3/3 2024-01-18T06:36:07.399 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: kernel-4.18.0-535.el8.x86_64 3/3 2024-01-18T06:36:07.399 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : kernel-4.18.0-535.el8.x86_64 1/3 2024-01-18T06:36:07.399 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : kernel-core-4.18.0-535.el8.x86_64 2/3 2024-01-18T06:36:07.951 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : kernel-modules-4.18.0-535.el8.x86_64 3/3 2024-01-18T06:36:07.951 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:36:07.951 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-01-18T06:36:07.951 INFO:teuthology.orchestra.run.smithi047.stdout: kernel-4.18.0-535.el8.x86_64 kernel-core-4.18.0-535.el8.x86_64 2024-01-18T06:36:07.951 INFO:teuthology.orchestra.run.smithi047.stdout: kernel-modules-4.18.0-535.el8.x86_64 2024-01-18T06:36:07.951 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:36:07.951 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-01-18T06:36:08.086 DEBUG:teuthology.orchestra.run.smithi047:> rpm -q kernel | sort -rV | head -n 1 2024-01-18T06:36:08.349 INFO:teuthology.orchestra.run.smithi047.stdout:kernel-4.18.0-535.el8.x86_64 2024-01-18T06:36:08.349 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-535.el8.x86_64 2024-01-18T06:36:08.349 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-535.el8.x86_64 2024-01-18T06:36:08.349 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-18T06:36:08.350 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-18T06:36:08.350 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-01-18T06:36:08.350 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi047.front.sepia.ceph.com, path=None, version=distro) 2024-01-18T06:36:08.350 DEBUG:teuthology.orchestra.run.smithi047:> rpm -q kernel | sort -rV | head -n 1 2024-01-18T06:36:08.569 INFO:teuthology.orchestra.run.smithi047.stdout:kernel-4.18.0-535.el8.x86_64 2024-01-18T06:36:08.570 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-535.el8.x86_64 2024-01-18T06:36:08.570 DEBUG:teuthology.orchestra.run.smithi047:> sudo rpm -qi grub2-tools 2024-01-18T06:36:08.605 INFO:teuthology.orchestra.run.smithi047.stdout:Name : grub2-tools 2024-01-18T06:36:08.605 INFO:teuthology.orchestra.run.smithi047.stdout:Epoch : 1 2024-01-18T06:36:08.605 INFO:teuthology.orchestra.run.smithi047.stdout:Version : 2.02 2024-01-18T06:36:08.605 INFO:teuthology.orchestra.run.smithi047.stdout:Release : 129.el8 2024-01-18T06:36:08.605 INFO:teuthology.orchestra.run.smithi047.stdout:Architecture: x86_64 2024-01-18T06:36:08.605 INFO:teuthology.orchestra.run.smithi047.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-01-18T06:36:08.605 INFO:teuthology.orchestra.run.smithi047.stdout:Group : System Environment/Base 2024-01-18T06:36:08.605 INFO:teuthology.orchestra.run.smithi047.stdout:Size : 9265586 2024-01-18T06:36:08.606 INFO:teuthology.orchestra.run.smithi047.stdout:License : GPLv3+ 2024-01-18T06:36:08.606 INFO:teuthology.orchestra.run.smithi047.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-01-18T06:36:08.606 INFO:teuthology.orchestra.run.smithi047.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-01-18T06:36:08.606 INFO:teuthology.orchestra.run.smithi047.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-01-18T06:36:08.606 INFO:teuthology.orchestra.run.smithi047.stdout:Build Host : kbuilder.bsys.centos.org 2024-01-18T06:36:08.606 INFO:teuthology.orchestra.run.smithi047.stdout:Relocations : (not relocatable) 2024-01-18T06:36:08.606 INFO:teuthology.orchestra.run.smithi047.stdout:Packager : CentOS BuildSystem 2024-01-18T06:36:08.606 INFO:teuthology.orchestra.run.smithi047.stdout:Vendor : CentOS 2024-01-18T06:36:08.606 INFO:teuthology.orchestra.run.smithi047.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-18T06:36:08.606 INFO:teuthology.orchestra.run.smithi047.stdout:Summary : Support tools for GRUB. 2024-01-18T06:36:08.606 INFO:teuthology.orchestra.run.smithi047.stdout:Description : 2024-01-18T06:36:08.606 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:36:08.606 INFO:teuthology.orchestra.run.smithi047.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-18T06:36:08.606 INFO:teuthology.orchestra.run.smithi047.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-18T06:36:08.606 INFO:teuthology.orchestra.run.smithi047.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-18T06:36:08.607 INFO:teuthology.orchestra.run.smithi047.stdout:hardware devices. 2024-01-18T06:36:08.607 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:36:08.607 INFO:teuthology.orchestra.run.smithi047.stdout:This subpackage provides tools for support of all platforms. 2024-01-18T06:36:08.608 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-18T06:36:08.609 INFO:teuthology.task.kernel:Updating grub on smithi047 to boot 4.18.0-535.el8.x86_64 2024-01-18T06:36:08.609 DEBUG:teuthology.orchestra.run.smithi047:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-18T06:36:10.101 INFO:teuthology.orchestra.run.smithi047.stderr:Generating grub configuration file ... 2024-01-18T06:36:12.511 INFO:teuthology.orchestra.run.smithi047.stderr:done 2024-01-18T06:36:12.512 DEBUG:teuthology.orchestra.run.smithi047:> mktemp 2024-01-18T06:36:12.542 INFO:teuthology.orchestra.run.smithi047.stdout:/tmp/tmp.s7WnrMVAJN 2024-01-18T06:36:12.542 DEBUG:teuthology.orchestra.run.smithi047:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.s7WnrMVAJN 2024-01-18T06:36:12.614 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 0666 /tmp/tmp.s7WnrMVAJN 2024-01-18T06:36:12.754 DEBUG:teuthology.orchestra.remote:smithi047:/tmp/tmp.s7WnrMVAJN is 6KB 2024-01-18T06:36:12.803 DEBUG:teuthology.orchestra.run.smithi047:> rm -fr /tmp/tmp.s7WnrMVAJN 2024-01-18T06:36:12.821 DEBUG:teuthology.orchestra.run.smithi047:> sudo /bin/ls /boot/loader/entries || true 2024-01-18T06:36:12.957 INFO:teuthology.orchestra.run.smithi047.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-01-18T06:36:12.957 INFO:teuthology.orchestra.run.smithi047.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-01-18T06:36:12.957 INFO:teuthology.orchestra.run.smithi047.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-01-18T06:36:12.957 INFO:teuthology.orchestra.run.smithi047.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-535.el8.x86_64.conf 2024-01-18T06:36:12.958 DEBUG:teuthology.orchestra.run.smithi047:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-535.el8.x86_64 2024-01-18T06:36:13.114 DEBUG:teuthology.orchestra.run.smithi047:> sudo shutdown -r now 2024-01-18T06:36:36.224 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-01-18T06:36:36.225 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-01-18T06:36:36.226 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-18T06:36:43.124 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-01-18T06:36:43.125 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-01-18T06:36:43.125 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-18T06:37:36.247 DEBUG:teuthology.orchestra.remote:timed out 2024-01-18T06:37:43.132 DEBUG:teuthology.orchestra.remote:timed out 2024-01-18T06:37:45.251 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-01-18T06:37:45.252 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-18T06:37:45.665 DEBUG:teuthology.orchestra.run.smithi194:> true 2024-01-18T06:37:46.022 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-01-18T06:37:46.022 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-535.el8.x86_64"... 2024-01-18T06:37:46.022 DEBUG:teuthology.orchestra.run.smithi194:> uname -r 2024-01-18T06:37:46.039 INFO:teuthology.orchestra.run.smithi194.stdout:4.18.0-535.el8.x86_64 2024-01-18T06:37:46.039 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-535.el8.x86_64 vs 4.18.0-535.el8.x86_64 2024-01-18T06:37:46.039 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-18T06:37:46.040 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-18T06:37:47.040 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-01-18T06:37:47.041 DEBUG:teuthology.orchestra.run.smithi194:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-18T06:37:47.119 INFO:teuthology.orchestra.run.smithi194.stdout:ttyS1 2024-01-18T06:37:47.138 DEBUG:teuthology.parallel:result is None 2024-01-18T06:37:52.133 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-01-18T06:37:52.134 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-18T06:37:52.687 DEBUG:teuthology.orchestra.run.smithi047:> true 2024-01-18T06:37:53.517 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-01-18T06:37:53.517 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-535.el8.x86_64"... 2024-01-18T06:37:53.517 DEBUG:teuthology.orchestra.run.smithi047:> uname -r 2024-01-18T06:37:54.035 INFO:teuthology.orchestra.run.smithi047.stdout:4.18.0-535.el8.x86_64 2024-01-18T06:37:54.035 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-535.el8.x86_64 vs 4.18.0-535.el8.x86_64 2024-01-18T06:37:54.035 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-18T06:37:54.035 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-18T06:37:55.036 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-01-18T06:37:55.037 DEBUG:teuthology.orchestra.run.smithi047:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-18T06:37:55.172 INFO:teuthology.orchestra.run.smithi047.stdout:ttyS1 2024-01-18T06:37:55.191 DEBUG:teuthology.parallel:result is None 2024-01-18T06:37:55.191 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-18T06:37:55.200 INFO:teuthology.task.internal:Creating test directory... 2024-01-18T06:37:55.201 DEBUG:teuthology.orchestra.run.smithi047:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-18T06:37:55.203 DEBUG:teuthology.orchestra.run.smithi194:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-18T06:37:55.226 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-18T06:37:55.233 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-18T06:37:55.243 INFO:teuthology.task.internal:Creating archive directory... 2024-01-18T06:37:55.244 DEBUG:teuthology.orchestra.run.smithi047:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-18T06:37:55.262 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-18T06:37:55.299 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-18T06:37:55.306 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-18T06:37:55.307 DEBUG:teuthology.orchestra.run.smithi047:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-18T06:37:55.324 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-18T06:37:55.369 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-18T06:37:55.376 INFO:teuthology.orchestra.run.smithi047.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-18T06:37:55.380 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-18T06:37:55.386 INFO:teuthology.orchestra.run.smithi047.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-18T06:37:55.387 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-18T06:37:55.398 INFO:teuthology.task.internal:Configuring sudo... 2024-01-18T06:37:55.398 DEBUG:teuthology.orchestra.run.smithi047:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-18T06:37:55.400 DEBUG:teuthology.orchestra.run.smithi194:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-18T06:37:55.451 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-18T06:37:55.459 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-18T06:37:55.460 DEBUG:teuthology.orchestra.run.smithi047:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-18T06:37:55.468 DEBUG:teuthology.orchestra.run.smithi194:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-18T06:37:55.510 DEBUG:teuthology.orchestra.run.smithi047:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-18T06:37:55.572 DEBUG:teuthology.orchestra.run.smithi047:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-18T06:37:55.623 DEBUG:teuthology.orchestra.run.smithi047:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-18T06:37:55.710 DEBUG:teuthology.orchestra.run.smithi047:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-18T06:37:55.751 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-01-18T06:37:55.751 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-18T06:37:55.818 DEBUG:teuthology.orchestra.run.smithi194:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-18T06:37:55.863 DEBUG:teuthology.orchestra.run.smithi194:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-18T06:37:55.918 DEBUG:teuthology.orchestra.run.smithi194:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-18T06:37:56.006 DEBUG:teuthology.orchestra.run.smithi194:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-18T06:37:56.049 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-01-18T06:37:56.049 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-18T06:37:56.117 DEBUG:teuthology.orchestra.run.smithi047:> sudo service rsyslog restart 2024-01-18T06:37:56.120 DEBUG:teuthology.orchestra.run.smithi194:> sudo service rsyslog restart 2024-01-18T06:37:56.175 INFO:teuthology.orchestra.run.smithi047.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-18T06:37:56.220 INFO:teuthology.orchestra.run.smithi194.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-18T06:37:56.571 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-18T06:37:56.579 INFO:teuthology.task.internal:Starting timer... 2024-01-18T06:37:56.579 INFO:teuthology.run_tasks:Running task pcp... 2024-01-18T06:37:56.675 INFO:teuthology.run_tasks:Running task selinux... 2024-01-18T06:37:56.683 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-01-18T06:37:56.684 DEBUG:teuthology.orchestra.run.smithi047:> sudo service auditd rotate 2024-01-18T06:37:56.777 INFO:teuthology.orchestra.run.smithi047.stdout:Rotating logs: 2024-01-18T06:37:56.779 DEBUG:teuthology.orchestra.run.smithi194:> sudo service auditd rotate 2024-01-18T06:37:56.852 INFO:teuthology.orchestra.run.smithi194.stdout:Rotating logs: 2024-01-18T06:37:56.854 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-18T06:37:56.854 DEBUG:teuthology.orchestra.run.smithi047:> /usr/sbin/getenforce 2024-01-18T06:37:56.921 INFO:teuthology.orchestra.run.smithi047.stdout:Permissive 2024-01-18T06:37:56.921 DEBUG:teuthology.orchestra.run.smithi194:> /usr/sbin/getenforce 2024-01-18T06:37:56.947 INFO:teuthology.orchestra.run.smithi194.stdout:Permissive 2024-01-18T06:37:56.948 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi047.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi194.front.sepia.ceph.com': 'permissive'} 2024-01-18T06:37:56.948 DEBUG:teuthology.orchestra.run.smithi047:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-18T06:37:56.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-18T06:37:56.976 DEBUG:teuthology.orchestra.run.smithi194:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-18T06:37:57.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-18T06:37:57.015 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-18T06:37:57.016 DEBUG:teuthology.orchestra.run.smithi047:> sudo /usr/sbin/setenforce permissive 2024-01-18T06:37:57.043 DEBUG:teuthology.orchestra.run.smithi194:> sudo /usr/sbin/setenforce permissive 2024-01-18T06:37:57.082 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-18T06:37:57.092 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-18T06:37:57.108 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-18T06:37:57.109 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi047.front.sepia.ceph.com,smithi194.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-18T06:43:39.082 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi047.front.sepia.ceph.com'), Remote(name='ubuntu@smithi194.front.sepia.ceph.com')] 2024-01-18T06:43:39.083 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-01-18T06:43:39.084 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-18T06:43:39.158 DEBUG:teuthology.orchestra.run.smithi047:> true 2024-01-18T06:43:39.228 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-01-18T06:43:39.228 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-01-18T06:43:39.229 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-18T06:43:39.300 DEBUG:teuthology.orchestra.run.smithi194:> true 2024-01-18T06:43:39.367 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-01-18T06:43:39.367 INFO:teuthology.run_tasks:Running task clock... 2024-01-18T06:43:39.391 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-18T06:43:39.391 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-18T06:43:39.392 DEBUG:teuthology.orchestra.run.smithi047:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-18T06:43:39.394 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-18T06:43:39.395 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-18T06:43:39.428 INFO:teuthology.orchestra.run.smithi047.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-18T06:43:39.443 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-18T06:43:39.444 INFO:teuthology.orchestra.run.smithi047.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-18T06:43:39.457 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-18T06:43:39.508 INFO:teuthology.orchestra.run.smithi047.stderr:sudo: ntpd: command not found 2024-01-18T06:43:39.519 INFO:teuthology.orchestra.run.smithi047.stdout:506 Cannot talk to daemon 2024-01-18T06:43:39.520 INFO:teuthology.orchestra.run.smithi194.stderr:sudo: ntpd: command not found 2024-01-18T06:43:39.531 INFO:teuthology.orchestra.run.smithi194.stdout:506 Cannot talk to daemon 2024-01-18T06:43:39.534 INFO:teuthology.orchestra.run.smithi047.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-18T06:43:39.546 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-18T06:43:39.548 INFO:teuthology.orchestra.run.smithi047.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-18T06:43:39.559 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-18T06:43:39.615 INFO:teuthology.orchestra.run.smithi047.stderr:bash: ntpq: command not found 2024-01-18T06:43:39.618 INFO:teuthology.orchestra.run.smithi047.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-18T06:43:39.618 INFO:teuthology.orchestra.run.smithi047.stdout:=============================================================================== 2024-01-18T06:43:39.618 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-18T06:43:39.618 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-18T06:43:39.618 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-18T06:43:39.618 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-18T06:43:39.631 INFO:teuthology.orchestra.run.smithi194.stderr:bash: ntpq: command not found 2024-01-18T06:43:39.634 INFO:teuthology.orchestra.run.smithi194.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-18T06:43:39.634 INFO:teuthology.orchestra.run.smithi194.stdout:=============================================================================== 2024-01-18T06:43:39.635 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-18T06:43:39.635 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-18T06:43:39.635 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-18T06:43:39.635 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-18T06:43:39.636 INFO:teuthology.run_tasks:Running task pexec... 2024-01-18T06:43:39.645 INFO:teuthology.task.pexec:Executing custom commands... 2024-01-18T06:43:39.645 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi047.front.sepia.ceph.com 2024-01-18T06:43:39.645 DEBUG:teuthology.orchestra.run.smithi047:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-18T06:43:39.645 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi194.front.sepia.ceph.com 2024-01-18T06:43:39.646 DEBUG:teuthology.orchestra.run.smithi194:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-18T06:43:40.434 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:29 ago on Thu 18 Jan 2024 06:42:11 AM UTC. 2024-01-18T06:43:40.447 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:19 ago on Thu 18 Jan 2024 06:42:21 AM UTC. 2024-01-18T06:43:42.361 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-18T06:43:42.362 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:43:42.362 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-01-18T06:43:42.362 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:43:42.362 INFO:teuthology.orchestra.run.smithi194.stdout:Resetting modules: 2024-01-18T06:43:42.362 INFO:teuthology.orchestra.run.smithi194.stdout: container-tools 2024-01-18T06:43:42.362 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:43:42.362 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-01-18T06:43:42.362 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:43:42.362 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:43:42.531 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-01-18T06:43:42.533 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:43:42.533 INFO:teuthology.orchestra.run.smithi047.stdout: Package Architecture Version Repository Size 2024-01-18T06:43:42.533 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:43:42.533 INFO:teuthology.orchestra.run.smithi047.stdout:Resetting modules: 2024-01-18T06:43:42.533 INFO:teuthology.orchestra.run.smithi047.stdout: container-tools 2024-01-18T06:43:42.533 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:43:42.533 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-01-18T06:43:42.533 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:43:42.533 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:43:42.626 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-18T06:43:42.799 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-01-18T06:43:43.437 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:32 ago on Thu 18 Jan 2024 06:42:11 AM UTC. 2024-01-18T06:43:43.641 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:22 ago on Thu 18 Jan 2024 06:42:21 AM UTC. 2024-01-18T06:43:45.368 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-18T06:43:45.403 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:43:45.404 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-01-18T06:43:45.404 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:43:45.404 INFO:teuthology.orchestra.run.smithi194.stdout:Installing group/module packages: 2024-01-18T06:43:45.404 INFO:teuthology.orchestra.run.smithi194.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2024-01-18T06:43:45.404 INFO:teuthology.orchestra.run.smithi194.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2024-01-18T06:43:45.404 INFO:teuthology.orchestra.run.smithi194.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2024-01-18T06:43:45.404 INFO:teuthology.orchestra.run.smithi194.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2024-01-18T06:43:45.404 INFO:teuthology.orchestra.run.smithi194.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2024-01-18T06:43:45.404 INFO:teuthology.orchestra.run.smithi194.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2024-01-18T06:43:45.405 INFO:teuthology.orchestra.run.smithi194.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2024-01-18T06:43:45.405 INFO:teuthology.orchestra.run.smithi194.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2024-01-18T06:43:45.405 INFO:teuthology.orchestra.run.smithi194.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2024-01-18T06:43:45.405 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-01-18T06:43:45.405 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-01-18T06:43:45.405 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-01-18T06:43:45.405 INFO:teuthology.orchestra.run.smithi194.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-01-18T06:43:45.405 INFO:teuthology.orchestra.run.smithi194.stdout:Installing weak dependencies: 2024-01-18T06:43:45.405 INFO:teuthology.orchestra.run.smithi194.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-01-18T06:43:45.405 INFO:teuthology.orchestra.run.smithi194.stdout:Installing module profiles: 2024-01-18T06:43:45.405 INFO:teuthology.orchestra.run.smithi194.stdout: container-tools/common 2024-01-18T06:43:45.405 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:43:45.405 INFO:teuthology.orchestra.run.smithi194.stdout:Enabling module streams: 2024-01-18T06:43:45.406 INFO:teuthology.orchestra.run.smithi194.stdout: container-tools rhel8 2024-01-18T06:43:45.406 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:43:45.406 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-01-18T06:43:45.406 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:43:45.406 INFO:teuthology.orchestra.run.smithi194.stdout:Install 13 Packages 2024-01-18T06:43:45.406 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:43:45.407 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 27 M 2024-01-18T06:43:45.407 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 89 M 2024-01-18T06:43:45.407 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-01-18T06:43:45.688 INFO:teuthology.orchestra.run.smithi194.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 7.4 MB/s | 750 kB 00:00 2024-01-18T06:43:45.722 INFO:teuthology.orchestra.run.smithi194.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 7.6 MB/s | 1.0 MB 00:00 2024-01-18T06:43:45.725 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-01-18T06:43:45.747 INFO:teuthology.orchestra.run.smithi194.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 647 kB/s | 38 kB 00:00 2024-01-18T06:43:45.763 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:43:45.763 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-01-18T06:43:45.763 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:43:45.763 INFO:teuthology.orchestra.run.smithi047.stdout:Installing group/module packages: 2024-01-18T06:43:45.763 INFO:teuthology.orchestra.run.smithi047.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2024-01-18T06:43:45.763 INFO:teuthology.orchestra.run.smithi047.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2024-01-18T06:43:45.763 INFO:teuthology.orchestra.run.smithi047.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2024-01-18T06:43:45.763 INFO:teuthology.orchestra.run.smithi047.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2024-01-18T06:43:45.763 INFO:teuthology.orchestra.run.smithi047.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2024-01-18T06:43:45.763 INFO:teuthology.orchestra.run.smithi047.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2024-01-18T06:43:45.763 INFO:teuthology.orchestra.run.smithi047.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2024-01-18T06:43:45.763 INFO:teuthology.orchestra.run.smithi047.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2024-01-18T06:43:45.763 INFO:teuthology.orchestra.run.smithi047.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2024-01-18T06:43:45.763 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-01-18T06:43:45.764 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-01-18T06:43:45.764 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-01-18T06:43:45.764 INFO:teuthology.orchestra.run.smithi047.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-01-18T06:43:45.764 INFO:teuthology.orchestra.run.smithi047.stdout:Installing weak dependencies: 2024-01-18T06:43:45.764 INFO:teuthology.orchestra.run.smithi047.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-01-18T06:43:45.764 INFO:teuthology.orchestra.run.smithi047.stdout:Installing module profiles: 2024-01-18T06:43:45.764 INFO:teuthology.orchestra.run.smithi047.stdout: container-tools/common 2024-01-18T06:43:45.764 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:43:45.764 INFO:teuthology.orchestra.run.smithi047.stdout:Enabling module streams: 2024-01-18T06:43:45.764 INFO:teuthology.orchestra.run.smithi047.stdout: container-tools rhel8 2024-01-18T06:43:45.764 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:43:45.764 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-01-18T06:43:45.764 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:43:45.764 INFO:teuthology.orchestra.run.smithi047.stdout:Install 13 Packages 2024-01-18T06:43:45.764 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:43:45.765 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 27 M 2024-01-18T06:43:45.765 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 89 M 2024-01-18T06:43:45.765 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-01-18T06:43:45.772 INFO:teuthology.orchestra.run.smithi194.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 4.7 MB/s | 241 kB 00:00 2024-01-18T06:43:45.806 INFO:teuthology.orchestra.run.smithi194.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 5.2 MB/s | 178 kB 00:00 2024-01-18T06:43:45.840 INFO:teuthology.orchestra.run.smithi194.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 743 kB/s | 25 kB 00:00 2024-01-18T06:43:45.898 INFO:teuthology.orchestra.run.smithi194.stdout:(7/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 26 MB/s | 3.9 MB 00:00 2024-01-18T06:43:45.910 INFO:teuthology.orchestra.run.smithi047.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 7.1 MB/s | 750 kB 00:00 2024-01-18T06:43:45.924 INFO:teuthology.orchestra.run.smithi194.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.1 MB/s | 94 kB 00:00 2024-01-18T06:43:45.944 INFO:teuthology.orchestra.run.smithi047.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 7.3 MB/s | 1.0 MB 00:00 2024-01-18T06:43:45.969 INFO:teuthology.orchestra.run.smithi047.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 647 kB/s | 38 kB 00:00 2024-01-18T06:43:46.028 INFO:teuthology.orchestra.run.smithi047.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 2.8 MB/s | 241 kB 00:00 2024-01-18T06:43:46.040 INFO:teuthology.orchestra.run.smithi194.stdout:(9/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 21 MB/s | 9.6 MB 00:00 2024-01-18T06:43:46.061 INFO:teuthology.orchestra.run.smithi047.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 5.3 MB/s | 178 kB 00:00 2024-01-18T06:43:46.095 INFO:teuthology.orchestra.run.smithi047.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 735 kB/s | 25 kB 00:00 2024-01-18T06:43:46.107 INFO:teuthology.orchestra.run.smithi194.stdout:(10/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 14 MB/s | 2.5 MB 00:00 2024-01-18T06:43:46.133 INFO:teuthology.orchestra.run.smithi194.stdout:(11/13): udica-0.2.6-20.module_el8+712+4cd1bd69 571 kB/s | 53 kB 00:00 2024-01-18T06:43:46.154 INFO:teuthology.orchestra.run.smithi047.stdout:(7/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 21 MB/s | 3.9 MB 00:00 2024-01-18T06:43:46.158 INFO:teuthology.orchestra.run.smithi194.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 809 kB/s | 41 kB 00:00 2024-01-18T06:43:46.179 INFO:teuthology.orchestra.run.smithi047.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.1 MB/s | 94 kB 00:00 2024-01-18T06:43:46.250 INFO:teuthology.orchestra.run.smithi194.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 23 MB/s | 8.1 MB 00:00 2024-01-18T06:43:46.251 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-01-18T06:43:46.251 INFO:teuthology.orchestra.run.smithi194.stdout:Total 31 MB/s | 27 MB 00:00 2024-01-18T06:43:46.330 INFO:teuthology.orchestra.run.smithi047.stdout:(9/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 18 MB/s | 9.6 MB 00:00 2024-01-18T06:43:46.371 INFO:teuthology.orchestra.run.smithi047.stdout:(10/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 13 MB/s | 2.5 MB 00:00 2024-01-18T06:43:46.397 INFO:teuthology.orchestra.run.smithi047.stdout:(11/13): udica-0.2.6-20.module_el8+712+4cd1bd69 789 kB/s | 53 kB 00:00 2024-01-18T06:43:46.414 INFO:teuthology.orchestra.run.smithi047.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 978 kB/s | 41 kB 00:00 2024-01-18T06:43:46.514 INFO:teuthology.orchestra.run.smithi047.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 23 MB/s | 8.1 MB 00:00 2024-01-18T06:43:46.514 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-01-18T06:43:46.514 INFO:teuthology.orchestra.run.smithi047.stdout:Total 35 MB/s | 27 MB 00:00 2024-01-18T06:43:46.515 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-01-18T06:43:46.536 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-01-18T06:43:46.536 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-01-18T06:43:46.789 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-01-18T06:43:46.793 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-01-18T06:43:46.796 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-01-18T06:43:46.810 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-01-18T06:43:46.810 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-01-18T06:43:47.089 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-01-18T06:43:47.092 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-01-18T06:43:47.464 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-01-18T06:43:47.721 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-01-18T06:43:47.823 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-01-18T06:43:47.883 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-01-18T06:43:48.069 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-01-18T06:43:48.183 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-01-18T06:43:48.310 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-18T06:43:48.394 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-01-18T06:43:48.628 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-01-18T06:43:48.843 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-18T06:43:48.858 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2024-01-18T06:43:49.403 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2024-01-18T06:43:49.856 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-18T06:43:50.087 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2024-01-18T06:43:50.291 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2024-01-18T06:43:50.429 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-18T06:43:50.571 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2024-01-18T06:43:50.743 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2024-01-18T06:43:50.927 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2024-01-18T06:43:50.967 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2024-01-18T06:43:51.256 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2024-01-18T06:43:51.695 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2024-01-18T06:43:51.824 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-18T06:43:52.029 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2024-01-18T06:43:52.199 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-18T06:43:52.634 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-18T06:43:52.634 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2024-01-18T06:43:52.634 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2024-01-18T06:43:52.634 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2024-01-18T06:43:52.634 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-18T06:43:52.634 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2024-01-18T06:43:52.634 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-18T06:43:52.634 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2024-01-18T06:43:52.634 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-01-18T06:43:52.634 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-01-18T06:43:52.634 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2024-01-18T06:43:52.634 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-18T06:43:52.635 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2024-01-18T06:43:52.663 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-18T06:43:52.907 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2024-01-18T06:43:53.116 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-18T06:43:53.158 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-01-18T06:43:53.158 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:43:53.158 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-01-18T06:43:53.158 INFO:teuthology.orchestra.run.smithi194.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-18T06:43:53.158 INFO:teuthology.orchestra.run.smithi194.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-18T06:43:53.158 INFO:teuthology.orchestra.run.smithi194.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2024-01-18T06:43:53.158 INFO:teuthology.orchestra.run.smithi194.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-01-18T06:43:53.159 INFO:teuthology.orchestra.run.smithi194.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2024-01-18T06:43:53.159 INFO:teuthology.orchestra.run.smithi194.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2024-01-18T06:43:53.159 INFO:teuthology.orchestra.run.smithi194.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2024-01-18T06:43:53.159 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-01-18T06:43:53.159 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-01-18T06:43:53.159 INFO:teuthology.orchestra.run.smithi194.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2024-01-18T06:43:53.159 INFO:teuthology.orchestra.run.smithi194.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2024-01-18T06:43:53.159 INFO:teuthology.orchestra.run.smithi194.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2024-01-18T06:43:53.159 INFO:teuthology.orchestra.run.smithi194.stdout: yajl-2.1.0-12.el8.x86_64 2024-01-18T06:43:53.159 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:43:53.159 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-18T06:43:53.418 DEBUG:teuthology.parallel:result is None 2024-01-18T06:43:53.580 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-18T06:43:53.580 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2024-01-18T06:43:53.580 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2024-01-18T06:43:53.580 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2024-01-18T06:43:53.580 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-18T06:43:53.580 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2024-01-18T06:43:53.580 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-18T06:43:53.580 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2024-01-18T06:43:53.580 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-01-18T06:43:53.580 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-01-18T06:43:53.581 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2024-01-18T06:43:53.581 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-18T06:43:53.581 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2024-01-18T06:43:54.084 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-01-18T06:43:54.084 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:43:54.085 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-01-18T06:43:54.085 INFO:teuthology.orchestra.run.smithi047.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-18T06:43:54.085 INFO:teuthology.orchestra.run.smithi047.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-18T06:43:54.085 INFO:teuthology.orchestra.run.smithi047.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2024-01-18T06:43:54.085 INFO:teuthology.orchestra.run.smithi047.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-01-18T06:43:54.085 INFO:teuthology.orchestra.run.smithi047.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2024-01-18T06:43:54.085 INFO:teuthology.orchestra.run.smithi047.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2024-01-18T06:43:54.085 INFO:teuthology.orchestra.run.smithi047.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2024-01-18T06:43:54.085 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-01-18T06:43:54.085 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-01-18T06:43:54.085 INFO:teuthology.orchestra.run.smithi047.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2024-01-18T06:43:54.085 INFO:teuthology.orchestra.run.smithi047.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2024-01-18T06:43:54.085 INFO:teuthology.orchestra.run.smithi047.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2024-01-18T06:43:54.085 INFO:teuthology.orchestra.run.smithi047.stdout: yajl-2.1.0-12.el8.x86_64 2024-01-18T06:43:54.085 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:43:54.086 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-01-18T06:43:54.398 DEBUG:teuthology.parallel:result is None 2024-01-18T06:43:54.398 INFO:teuthology.run_tasks:Running task pexec... 2024-01-18T06:43:54.407 INFO:teuthology.task.pexec:Executing custom commands... 2024-01-18T06:43:54.407 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi047.front.sepia.ceph.com 2024-01-18T06:43:54.407 DEBUG:teuthology.orchestra.run.smithi047:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-18T06:43:54.408 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi194.front.sepia.ceph.com 2024-01-18T06:43:54.408 DEBUG:teuthology.orchestra.run.smithi194:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-18T06:43:54.580 DEBUG:teuthology.parallel:result is None 2024-01-18T06:43:54.601 DEBUG:teuthology.parallel:result is None 2024-01-18T06:43:54.601 INFO:teuthology.run_tasks:Running task install... 2024-01-18T06:43:54.611 DEBUG:teuthology.task.install:project ceph 2024-01-18T06:43:54.611 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0'}} 2024-01-18T06:43:54.611 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': '491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0'} 2024-01-18T06:43:54.611 INFO:teuthology.task.install:Using flavor: default 2024-01-18T06:43:54.619 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-01-18T06:43:54.619 INFO:teuthology.task.install:extra packages: [] 2024-01-18T06:43:54.619 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-01-18T06:43:54.620 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-18T06:43:54.620 INFO:teuthology.packaging:ref: None 2024-01-18T06:43:54.620 INFO:teuthology.packaging:tag: v16.2.4 2024-01-18T06:43:54.620 INFO:teuthology.packaging:branch: None 2024-01-18T06:43:54.620 INFO:teuthology.packaging:sha1: 491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0 2024-01-18T06:43:54.883 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-18T06:43:54.883 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-18T06:43:54.886 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-01-18T06:43:54.886 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-18T06:43:54.887 INFO:teuthology.packaging:ref: None 2024-01-18T06:43:54.887 INFO:teuthology.packaging:tag: v16.2.4 2024-01-18T06:43:54.887 INFO:teuthology.packaging:branch: None 2024-01-18T06:43:54.887 INFO:teuthology.packaging:sha1: 491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0 2024-01-18T06:43:54.887 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-18T06:43:55.033 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-01-18T06:43:55.033 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-01-18T06:43:55.046 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-01-18T06:43:55.046 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-01-18T06:43:55.084 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-18T06:43:55.084 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-01-18T06:43:55.084 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-18T06:43:55.097 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-18T06:43:55.097 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-01-18T06:43:55.097 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-18T06:43:55.116 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-01-18T06:43:55.116 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-18T06:43:55.116 INFO:teuthology.packaging:ref: None 2024-01-18T06:43:55.116 INFO:teuthology.packaging:tag: v16.2.4 2024-01-18T06:43:55.116 INFO:teuthology.packaging:branch: None 2024-01-18T06:43:55.116 INFO:teuthology.packaging:sha1: 491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0 2024-01-18T06:43:55.116 DEBUG:teuthology.orchestra.run.smithi047:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-18T06:43:55.127 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-01-18T06:43:55.127 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-18T06:43:55.127 INFO:teuthology.packaging:ref: None 2024-01-18T06:43:55.127 INFO:teuthology.packaging:tag: v16.2.4 2024-01-18T06:43:55.127 INFO:teuthology.packaging:branch: None 2024-01-18T06:43:55.127 INFO:teuthology.packaging:sha1: 491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0 2024-01-18T06:43:55.127 DEBUG:teuthology.orchestra.run.smithi194:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-18T06:43:55.187 DEBUG:teuthology.orchestra.run.smithi047:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-18T06:43:55.196 DEBUG:teuthology.orchestra.run.smithi194:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-18T06:43:55.269 DEBUG:teuthology.orchestra.run.smithi047:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-18T06:43:55.274 DEBUG:teuthology.orchestra.run.smithi194:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-18T06:43:55.338 INFO:teuthology.orchestra.run.smithi047.stdout:check_obsoletes = 1 2024-01-18T06:43:55.339 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum clean all 2024-01-18T06:43:55.341 INFO:teuthology.orchestra.run.smithi194.stdout:check_obsoletes = 1 2024-01-18T06:43:55.342 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum clean all 2024-01-18T06:43:55.607 INFO:teuthology.orchestra.run.smithi194.stdout:76 files removed 2024-01-18T06:43:55.631 INFO:teuthology.orchestra.run.smithi047.stdout:76 files removed 2024-01-18T06:43:55.634 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-radosgw 2024-01-18T06:43:55.662 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-radosgw 2024-01-18T06:43:56.107 INFO:teuthology.orchestra.run.smithi194.stdout:ceph packages for x86_64 1.1 MB/s | 74 kB 00:00 2024-01-18T06:43:56.175 INFO:teuthology.orchestra.run.smithi047.stdout:ceph packages for x86_64 934 kB/s | 74 kB 00:00 2024-01-18T06:43:56.225 INFO:teuthology.orchestra.run.smithi194.stdout:ceph noarch packages 190 kB/s | 15 kB 00:00 2024-01-18T06:43:56.292 INFO:teuthology.orchestra.run.smithi047.stdout:ceph noarch packages 199 kB/s | 15 kB 00:00 2024-01-18T06:43:56.325 INFO:teuthology.orchestra.run.smithi194.stdout:ceph source packages 24 kB/s | 1.7 kB 00:00 2024-01-18T06:43:56.401 INFO:teuthology.orchestra.run.smithi047.stdout:ceph source packages 22 kB/s | 1.7 kB 00:00 2024-01-18T06:43:56.851 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 8 - AppStream 53 MB/s | 26 MB 00:00 2024-01-18T06:43:57.127 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 8 - AppStream 38 MB/s | 26 MB 00:00 2024-01-18T06:44:01.993 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 8 - BaseOS 26 MB/s | 10 MB 00:00 2024-01-18T06:44:03.161 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 8 - BaseOS 9.9 MB/s | 10 MB 00:01 2024-01-18T06:44:04.210 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 8 - Extras 29 kB/s | 18 kB 00:00 2024-01-18T06:44:05.117 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 8 - Extras 64 kB/s | 18 kB 00:00 2024-01-18T06:44:06.077 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 8 - Extras common packages 3.9 kB/s | 6.9 kB 00:01 2024-01-18T06:44:06.313 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 8 - Extras common packages 6.4 kB/s | 6.9 kB 00:01 2024-01-18T06:44:06.970 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 8 - PowerTools 5.9 MB/s | 4.7 MB 00:00 2024-01-18T06:44:07.489 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 8 - PowerTools 4.4 MB/s | 4.7 MB 00:01 2024-01-18T06:44:07.972 INFO:teuthology.orchestra.run.smithi194.stdout:Copr repo for python3-asyncssh owned by ceph 31 kB/s | 3.5 kB 00:00 2024-01-18T06:44:08.882 INFO:teuthology.orchestra.run.smithi047.stdout:Copr repo for python3-asyncssh owned by ceph 7.6 kB/s | 3.5 kB 00:00 2024-01-18T06:44:10.199 INFO:teuthology.orchestra.run.smithi047.stdout:Extra Packages for Enterprise Linux 13 MB/s | 16 MB 00:01 2024-01-18T06:44:14.191 INFO:teuthology.orchestra.run.smithi047.stdout:lab-extras 411 kB/s | 24 kB 00:00 2024-01-18T06:44:14.672 INFO:teuthology.orchestra.run.smithi194.stdout:Extra Packages for Enterprise Linux 2.4 MB/s | 16 MB 00:06 2024-01-18T06:44:17.863 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-01-18T06:44:17.866 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:44:17.866 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-01-18T06:44:17.866 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:44:17.866 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-01-18T06:44:17.866 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-01-18T06:44:17.867 INFO:teuthology.orchestra.run.smithi047.stdout:Upgrading: 2024-01-18T06:44:17.867 INFO:teuthology.orchestra.run.smithi047.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-18T06:44:17.867 INFO:teuthology.orchestra.run.smithi047.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-01-18T06:44:17.867 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-01-18T06:44:17.867 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-01-18T06:44:17.867 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-01-18T06:44:17.867 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-01-18T06:44:17.867 INFO:teuthology.orchestra.run.smithi047.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-01-18T06:44:17.867 INFO:teuthology.orchestra.run.smithi047.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-01-18T06:44:17.867 INFO:teuthology.orchestra.run.smithi047.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-18T06:44:17.867 INFO:teuthology.orchestra.run.smithi047.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-01-18T06:44:17.867 INFO:teuthology.orchestra.run.smithi047.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-01-18T06:44:17.867 INFO:teuthology.orchestra.run.smithi047.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-18T06:44:17.867 INFO:teuthology.orchestra.run.smithi047.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-01-18T06:44:17.868 INFO:teuthology.orchestra.run.smithi047.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-01-18T06:44:17.868 INFO:teuthology.orchestra.run.smithi047.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-01-18T06:44:17.868 INFO:teuthology.orchestra.run.smithi047.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-01-18T06:44:17.868 INFO:teuthology.orchestra.run.smithi047.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-01-18T06:44:17.868 INFO:teuthology.orchestra.run.smithi047.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-18T06:44:17.868 INFO:teuthology.orchestra.run.smithi047.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-01-18T06:44:17.868 INFO:teuthology.orchestra.run.smithi047.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-01-18T06:44:17.868 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-01-18T06:44:17.868 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-01-18T06:44:17.868 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-01-18T06:44:17.868 INFO:teuthology.orchestra.run.smithi047.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-01-18T06:44:17.868 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-01-18T06:44:17.868 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-01-18T06:44:17.868 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-01-18T06:44:17.868 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:44:17.869 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-01-18T06:44:17.869 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:44:17.869 INFO:teuthology.orchestra.run.smithi047.stdout:Install 25 Packages 2024-01-18T06:44:17.869 INFO:teuthology.orchestra.run.smithi047.stdout:Upgrade 2 Packages 2024-01-18T06:44:17.869 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:44:17.869 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 59 M 2024-01-18T06:44:17.869 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-01-18T06:44:18.421 INFO:teuthology.orchestra.run.smithi194.stdout:lab-extras 413 kB/s | 24 kB 00:00 2024-01-18T06:44:18.509 INFO:teuthology.orchestra.run.smithi047.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 41 MB/s | 6.0 MB 00:00 2024-01-18T06:44:18.534 INFO:teuthology.orchestra.run.smithi047.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 993 kB/s | 25 kB 00:00 2024-01-18T06:44:18.576 INFO:teuthology.orchestra.run.smithi047.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 19 MB/s | 801 kB 00:00 2024-01-18T06:44:18.641 INFO:teuthology.orchestra.run.smithi047.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 7.3 MB/s | 486 kB 00:00 2024-01-18T06:44:18.743 INFO:teuthology.orchestra.run.smithi047.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 40 MB/s | 4.1 MB 00:00 2024-01-18T06:44:18.768 INFO:teuthology.orchestra.run.smithi047.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.7 MB/s | 44 kB 00:00 2024-01-18T06:44:18.794 INFO:teuthology.orchestra.run.smithi047.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2024-01-18T06:44:18.828 INFO:teuthology.orchestra.run.smithi047.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 6.2 MB/s | 213 kB 00:00 2024-01-18T06:44:18.863 INFO:teuthology.orchestra.run.smithi047.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2024-01-18T06:44:18.920 INFO:teuthology.orchestra.run.smithi047.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 6.2 MB/s | 368 kB 00:00 2024-01-18T06:44:18.954 INFO:teuthology.orchestra.run.smithi047.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 3.3 MB/s | 114 kB 00:00 2024-01-18T06:44:19.107 INFO:teuthology.orchestra.run.smithi047.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 17 MB/s | 13 MB 00:00 2024-01-18T06:44:19.129 INFO:teuthology.orchestra.run.smithi047.stdout:(13/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 829 kB/s | 145 kB 00:00 2024-01-18T06:44:19.206 INFO:teuthology.orchestra.run.smithi047.stdout:(14/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 3.3 MB/s | 259 kB 00:00 2024-01-18T06:44:19.239 INFO:teuthology.orchestra.run.smithi047.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 1.4 MB/s | 44 kB 00:00 2024-01-18T06:44:19.497 INFO:teuthology.orchestra.run.smithi047.stdout:(16/27): ceph-common-16.2.4-0.el8.x86_64.rpm 21 MB/s | 24 MB 00:01 2024-01-18T06:44:19.531 INFO:teuthology.orchestra.run.smithi047.stdout:(17/27): librdkafka-1.6.1-1.el8.x86_64.rpm 1.6 MB/s | 677 kB 00:00 2024-01-18T06:44:19.556 INFO:teuthology.orchestra.run.smithi047.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 133 kB/s | 42 kB 00:00 2024-01-18T06:44:19.606 INFO:teuthology.orchestra.run.smithi047.stdout:(19/27): ndctl-libs-71.1-7.el8.x86_64.rpm 1.6 MB/s | 80 kB 00:00 2024-01-18T06:44:19.632 INFO:teuthology.orchestra.run.smithi047.stdout:(20/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 351 kB/s | 47 kB 00:00 2024-01-18T06:44:19.957 INFO:teuthology.orchestra.run.smithi047.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 290 kB/s | 116 kB 00:00 2024-01-18T06:44:20.057 INFO:teuthology.orchestra.run.smithi047.stdout:(22/27): liboath-2.6.2-3.el8.x86_64.rpm 589 kB/s | 59 kB 00:00 2024-01-18T06:44:20.082 INFO:teuthology.orchestra.run.smithi047.stdout:(23/27): leveldb-1.22-1.el8.x86_64.rpm 402 kB/s | 181 kB 00:00 2024-01-18T06:44:20.107 INFO:teuthology.orchestra.run.smithi047.stdout:(24/27): gperftools-libs-2.7-9.el8.x86_64.rpm 611 kB/s | 306 kB 00:00 2024-01-18T06:44:20.150 INFO:teuthology.orchestra.run.smithi047.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 813 kB/s | 75 kB 00:00 2024-01-18T06:44:20.216 INFO:teuthology.orchestra.run.smithi047.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 28 MB/s | 3.7 MB 00:00 2024-01-18T06:44:20.275 INFO:teuthology.orchestra.run.smithi047.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 24 MB/s | 4.0 MB 00:00 2024-01-18T06:44:20.277 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-01-18T06:44:20.277 INFO:teuthology.orchestra.run.smithi047.stdout:Total 24 MB/s | 59 MB 00:02 2024-01-18T06:44:20.349 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-01-18T06:44:20.402 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-01-18T06:44:20.402 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-01-18T06:44:20.782 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-01-18T06:44:20.785 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-01-18T06:44:21.926 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-18T06:44:21.929 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:44:21.929 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-01-18T06:44:21.929 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:44:21.929 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-01-18T06:44:21.929 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-01-18T06:44:21.929 INFO:teuthology.orchestra.run.smithi194.stdout:Upgrading: 2024-01-18T06:44:21.929 INFO:teuthology.orchestra.run.smithi194.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-18T06:44:21.929 INFO:teuthology.orchestra.run.smithi194.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-01-18T06:44:21.929 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-01-18T06:44:21.929 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-01-18T06:44:21.929 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-01-18T06:44:21.930 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-01-18T06:44:21.930 INFO:teuthology.orchestra.run.smithi194.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-01-18T06:44:21.930 INFO:teuthology.orchestra.run.smithi194.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-01-18T06:44:21.930 INFO:teuthology.orchestra.run.smithi194.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-18T06:44:21.930 INFO:teuthology.orchestra.run.smithi194.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-01-18T06:44:21.930 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-01-18T06:44:21.930 INFO:teuthology.orchestra.run.smithi194.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-18T06:44:21.930 INFO:teuthology.orchestra.run.smithi194.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-01-18T06:44:21.930 INFO:teuthology.orchestra.run.smithi194.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-01-18T06:44:21.930 INFO:teuthology.orchestra.run.smithi194.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-01-18T06:44:21.930 INFO:teuthology.orchestra.run.smithi194.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-01-18T06:44:21.930 INFO:teuthology.orchestra.run.smithi194.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-01-18T06:44:21.930 INFO:teuthology.orchestra.run.smithi194.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-18T06:44:21.930 INFO:teuthology.orchestra.run.smithi194.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-01-18T06:44:21.930 INFO:teuthology.orchestra.run.smithi194.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-01-18T06:44:21.930 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-01-18T06:44:21.931 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-01-18T06:44:21.931 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-01-18T06:44:21.931 INFO:teuthology.orchestra.run.smithi194.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-01-18T06:44:21.931 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-01-18T06:44:21.931 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-01-18T06:44:21.931 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-01-18T06:44:21.931 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:44:21.931 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-01-18T06:44:21.931 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:44:21.931 INFO:teuthology.orchestra.run.smithi194.stdout:Install 25 Packages 2024-01-18T06:44:21.931 INFO:teuthology.orchestra.run.smithi194.stdout:Upgrade 2 Packages 2024-01-18T06:44:21.931 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:44:21.932 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 59 M 2024-01-18T06:44:21.932 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-01-18T06:44:22.415 INFO:teuthology.orchestra.run.smithi194.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 43 MB/s | 6.0 MB 00:00 2024-01-18T06:44:22.440 INFO:teuthology.orchestra.run.smithi194.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 996 kB/s | 25 kB 00:00 2024-01-18T06:44:22.473 INFO:teuthology.orchestra.run.smithi194.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 24 MB/s | 801 kB 00:00 2024-01-18T06:44:22.507 INFO:teuthology.orchestra.run.smithi194.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2024-01-18T06:44:22.591 INFO:teuthology.orchestra.run.smithi194.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 49 MB/s | 4.1 MB 00:00 2024-01-18T06:44:22.616 INFO:teuthology.orchestra.run.smithi194.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.8 MB/s | 44 kB 00:00 2024-01-18T06:44:22.641 INFO:teuthology.orchestra.run.smithi194.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2024-01-18T06:44:22.666 INFO:teuthology.orchestra.run.smithi194.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.3 MB/s | 213 kB 00:00 2024-01-18T06:44:22.700 INFO:teuthology.orchestra.run.smithi194.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2024-01-18T06:44:22.725 INFO:teuthology.orchestra.run.smithi194.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 14 MB/s | 368 kB 00:00 2024-01-18T06:44:22.742 INFO:teuthology.orchestra.run.smithi194.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 6.7 MB/s | 114 kB 00:00 2024-01-18T06:44:22.852 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-01-18T06:44:22.893 INFO:teuthology.orchestra.run.smithi194.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 20 MB/s | 13 MB 00:00 2024-01-18T06:44:22.951 INFO:teuthology.orchestra.run.smithi194.stdout:(13/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 695 kB/s | 145 kB 00:00 2024-01-18T06:44:23.100 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-01-18T06:44:23.168 INFO:teuthology.orchestra.run.smithi194.stdout:(14/27): ceph-common-16.2.4-0.el8.x86_64.rpm 27 MB/s | 24 MB 00:00 2024-01-18T06:44:23.193 INFO:teuthology.orchestra.run.smithi194.stdout:(15/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.0 MB/s | 259 kB 00:00 2024-01-18T06:44:23.227 INFO:teuthology.orchestra.run.smithi194.stdout:(16/27): librdkafka-1.6.1-1.el8.x86_64.rpm 2.0 MB/s | 677 kB 00:00 2024-01-18T06:44:23.244 INFO:teuthology.orchestra.run.smithi194.stdout:(17/27): python3-prettytable-0.7.2-14.el8.noarc 583 kB/s | 44 kB 00:00 2024-01-18T06:44:23.289 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-01-18T06:44:23.294 INFO:teuthology.orchestra.run.smithi194.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 419 kB/s | 42 kB 00:00 2024-01-18T06:44:23.320 INFO:teuthology.orchestra.run.smithi194.stdout:(19/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 510 kB/s | 47 kB 00:00 2024-01-18T06:44:23.337 INFO:teuthology.orchestra.run.smithi194.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 867 kB/s | 80 kB 00:00 2024-01-18T06:44:23.471 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-18T06:44:23.729 INFO:teuthology.orchestra.run.smithi194.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 268 kB/s | 116 kB 00:00 2024-01-18T06:44:23.754 INFO:teuthology.orchestra.run.smithi194.stdout:(22/27): leveldb-1.22-1.el8.x86_64.rpm 434 kB/s | 181 kB 00:00 2024-01-18T06:44:23.796 INFO:teuthology.orchestra.run.smithi194.stdout:(23/27): gperftools-libs-2.7-9.el8.x86_64.rpm 644 kB/s | 306 kB 00:00 2024-01-18T06:44:23.830 INFO:teuthology.orchestra.run.smithi194.stdout:(24/27): liboath-2.6.2-3.el8.x86_64.rpm 585 kB/s | 59 kB 00:00 2024-01-18T06:44:23.855 INFO:teuthology.orchestra.run.smithi194.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 747 kB/s | 75 kB 00:00 2024-01-18T06:44:23.897 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-18T06:44:23.938 INFO:teuthology.orchestra.run.smithi194.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 26 MB/s | 3.7 MB 00:00 2024-01-18T06:44:23.997 INFO:teuthology.orchestra.run.smithi194.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 24 MB/s | 4.0 MB 00:00 2024-01-18T06:44:23.999 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-01-18T06:44:23.999 INFO:teuthology.orchestra.run.smithi194.stdout:Total 29 MB/s | 59 MB 00:02 2024-01-18T06:44:24.069 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-01-18T06:44:24.118 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-01-18T06:44:24.118 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-01-18T06:44:24.137 INFO:teuthology.orchestra.run.smithi047.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-18T06:44:24.236 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-18T06:44:24.459 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-01-18T06:44:24.462 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-01-18T06:44:24.506 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-01-18T06:44:24.646 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-18T06:44:24.722 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-18T06:44:24.943 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-01-18T06:44:25.568 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-01-18T06:44:25.620 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-01-18T06:44:25.770 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-18T06:44:25.799 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-01-18T06:44:25.887 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-18T06:44:26.015 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-01-18T06:44:26.059 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-01-18T06:44:26.237 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-18T06:44:26.302 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-18T06:44:26.511 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-01-18T06:44:26.669 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-18T06:44:26.722 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-01-18T06:44:26.879 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-18T06:44:26.980 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-18T06:44:27.088 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-18T06:44:27.147 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-18T06:44:27.294 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-18T06:44:27.392 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-18T06:44:27.609 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-01-18T06:44:27.633 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-18T06:44:27.752 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-18T06:44:27.818 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-18T06:44:27.829 INFO:teuthology.orchestra.run.smithi047.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-18T06:44:27.916 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-18T06:44:28.056 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-01-18T06:44:28.241 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-01-18T06:44:28.475 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-01-18T06:44:28.577 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-01-18T06:44:28.637 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-18T06:44:28.705 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-18T06:44:28.726 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-18T06:44:28.807 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-18T06:44:28.896 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-01-18T06:44:28.998 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-01-18T06:44:29.063 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-01-18T06:44:29.152 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-18T06:44:29.224 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-18T06:44:29.258 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-01-18T06:44:29.416 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-01-18T06:44:29.479 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-01-18T06:44:29.594 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-01-18T06:44:29.760 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-18T06:44:29.834 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-18T06:44:29.994 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-18T06:44:30.458 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-18T06:44:30.609 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-18T06:44:30.711 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-18T06:44:30.880 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-01-18T06:44:31.070 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-01-18T06:44:31.193 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-18T06:44:31.267 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-18T06:44:31.443 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-01-18T06:44:31.827 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-01-18T06:44:32.011 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-01-18T06:44:32.210 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-18T06:44:32.243 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-01-18T06:44:32.437 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-18T06:44:33.226 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-18T06:44:33.533 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-18T06:44:33.684 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-18T06:44:33.844 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-18T06:44:34.767 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-18T06:44:35.010 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-18T06:44:35.707 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-18T06:44:35.992 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-18T06:44:36.099 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-18T06:44:36.234 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-18T06:44:53.878 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-18T06:44:53.879 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /sys 2024-01-18T06:44:53.879 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /proc 2024-01-18T06:44:53.879 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /dev 2024-01-18T06:44:53.879 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /run 2024-01-18T06:44:53.879 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /mnt 2024-01-18T06:44:53.879 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /var/tmp 2024-01-18T06:44:53.879 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /home 2024-01-18T06:44:53.879 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /tmp 2024-01-18T06:44:53.879 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /dev 2024-01-18T06:44:53.879 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:44:55.288 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-18T06:44:55.387 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-18T06:44:55.387 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-18T06:44:55.455 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-18T06:44:55.456 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /sys 2024-01-18T06:44:55.456 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /proc 2024-01-18T06:44:55.456 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /dev 2024-01-18T06:44:55.456 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /run 2024-01-18T06:44:55.456 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /mnt 2024-01-18T06:44:55.456 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /var/tmp 2024-01-18T06:44:55.456 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /home 2024-01-18T06:44:55.456 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /tmp 2024-01-18T06:44:55.456 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /dev 2024-01-18T06:44:55.456 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:44:55.578 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-18T06:44:55.578 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-18T06:44:56.455 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-18T06:44:56.456 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-01-18T06:44:56.456 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-01-18T06:44:56.456 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-01-18T06:44:56.456 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-01-18T06:44:56.456 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-01-18T06:44:56.456 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-01-18T06:44:56.456 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-01-18T06:44:56.456 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-01-18T06:44:56.456 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-01-18T06:44:56.456 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-01-18T06:44:56.456 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-01-18T06:44:56.457 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-01-18T06:44:56.457 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-01-18T06:44:56.457 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-01-18T06:44:56.457 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-01-18T06:44:56.457 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-01-18T06:44:56.457 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-01-18T06:44:56.457 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-01-18T06:44:56.457 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-01-18T06:44:56.457 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-01-18T06:44:56.457 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-01-18T06:44:56.457 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-01-18T06:44:56.457 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-01-18T06:44:56.457 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-01-18T06:44:56.457 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-01-18T06:44:56.457 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-01-18T06:44:56.457 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-01-18T06:44:56.458 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-01-18T06:44:56.789 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-18T06:44:56.863 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-18T06:44:56.863 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-18T06:44:57.074 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-18T06:44:57.075 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-18T06:44:57.475 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-01-18T06:44:57.475 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:44:57.475 INFO:teuthology.orchestra.run.smithi047.stdout:Upgraded: 2024-01-18T06:44:57.475 INFO:teuthology.orchestra.run.smithi047.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-01-18T06:44:57.475 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-01-18T06:44:57.475 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-01-18T06:44:57.475 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-01-18T06:44:57.475 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-01-18T06:44:57.475 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-01-18T06:44:57.476 INFO:teuthology.orchestra.run.smithi047.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-01-18T06:44:57.476 INFO:teuthology.orchestra.run.smithi047.stdout: fmt-6.2.1-1.el8.x86_64 2024-01-18T06:44:57.476 INFO:teuthology.orchestra.run.smithi047.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-18T06:44:57.476 INFO:teuthology.orchestra.run.smithi047.stdout: leveldb-1.22-1.el8.x86_64 2024-01-18T06:44:57.476 INFO:teuthology.orchestra.run.smithi047.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-01-18T06:44:57.476 INFO:teuthology.orchestra.run.smithi047.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-18T06:44:57.476 INFO:teuthology.orchestra.run.smithi047.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-01-18T06:44:57.476 INFO:teuthology.orchestra.run.smithi047.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-01-18T06:44:57.476 INFO:teuthology.orchestra.run.smithi047.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-01-18T06:44:57.476 INFO:teuthology.orchestra.run.smithi047.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-01-18T06:44:57.476 INFO:teuthology.orchestra.run.smithi047.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-01-18T06:44:57.476 INFO:teuthology.orchestra.run.smithi047.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-18T06:44:57.476 INFO:teuthology.orchestra.run.smithi047.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-18T06:44:57.477 INFO:teuthology.orchestra.run.smithi047.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-01-18T06:44:57.477 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-01-18T06:44:57.477 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-01-18T06:44:57.477 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-01-18T06:44:57.477 INFO:teuthology.orchestra.run.smithi047.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-18T06:44:57.477 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-01-18T06:44:57.477 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-01-18T06:44:57.477 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-01-18T06:44:57.477 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:44:57.477 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-01-18T06:44:57.761 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-test 2024-01-18T06:44:57.793 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-18T06:44:57.793 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-01-18T06:44:57.793 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-01-18T06:44:57.793 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-01-18T06:44:57.794 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-01-18T06:44:57.794 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-01-18T06:44:57.794 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-01-18T06:44:57.794 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-01-18T06:44:57.794 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-01-18T06:44:57.794 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-01-18T06:44:57.794 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-01-18T06:44:57.794 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-01-18T06:44:57.794 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-01-18T06:44:57.794 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-01-18T06:44:57.794 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-01-18T06:44:57.794 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-01-18T06:44:57.794 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-01-18T06:44:57.794 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-01-18T06:44:57.795 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-01-18T06:44:57.795 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-01-18T06:44:57.795 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-01-18T06:44:57.795 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-01-18T06:44:57.795 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-01-18T06:44:57.795 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-01-18T06:44:57.795 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-01-18T06:44:57.795 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-01-18T06:44:57.795 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-01-18T06:44:57.795 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-01-18T06:44:57.795 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-01-18T06:44:58.519 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:00:44 ago on Thu 18 Jan 2024 06:44:14 AM UTC. 2024-01-18T06:44:58.779 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-01-18T06:44:58.779 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:44:58.779 INFO:teuthology.orchestra.run.smithi194.stdout:Upgraded: 2024-01-18T06:44:58.779 INFO:teuthology.orchestra.run.smithi194.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-01-18T06:44:58.780 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-01-18T06:44:58.780 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-01-18T06:44:58.780 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-01-18T06:44:58.780 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-01-18T06:44:58.780 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-01-18T06:44:58.780 INFO:teuthology.orchestra.run.smithi194.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-01-18T06:44:58.780 INFO:teuthology.orchestra.run.smithi194.stdout: fmt-6.2.1-1.el8.x86_64 2024-01-18T06:44:58.780 INFO:teuthology.orchestra.run.smithi194.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-18T06:44:58.780 INFO:teuthology.orchestra.run.smithi194.stdout: leveldb-1.22-1.el8.x86_64 2024-01-18T06:44:58.780 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-01-18T06:44:58.780 INFO:teuthology.orchestra.run.smithi194.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-18T06:44:58.780 INFO:teuthology.orchestra.run.smithi194.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-01-18T06:44:58.780 INFO:teuthology.orchestra.run.smithi194.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-01-18T06:44:58.781 INFO:teuthology.orchestra.run.smithi194.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-01-18T06:44:58.781 INFO:teuthology.orchestra.run.smithi194.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-01-18T06:44:58.781 INFO:teuthology.orchestra.run.smithi194.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-01-18T06:44:58.781 INFO:teuthology.orchestra.run.smithi194.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-18T06:44:58.781 INFO:teuthology.orchestra.run.smithi194.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-18T06:44:58.781 INFO:teuthology.orchestra.run.smithi194.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-01-18T06:44:58.781 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-01-18T06:44:58.781 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-01-18T06:44:58.781 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-01-18T06:44:58.781 INFO:teuthology.orchestra.run.smithi194.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-18T06:44:58.781 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-01-18T06:44:58.781 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-01-18T06:44:58.781 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-01-18T06:44:58.781 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:44:58.781 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-18T06:44:59.053 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-test 2024-01-18T06:44:59.783 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:00:41 ago on Thu 18 Jan 2024 06:44:18 AM UTC. 2024-01-18T06:45:00.115 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-01-18T06:45:00.116 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:45:00.116 INFO:teuthology.orchestra.run.smithi047.stdout: Package Architecture Version Repository Size 2024-01-18T06:45:00.116 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:45:00.116 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-01-18T06:45:00.116 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-01-18T06:45:00.116 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-01-18T06:45:00.117 INFO:teuthology.orchestra.run.smithi047.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-01-18T06:45:00.117 INFO:teuthology.orchestra.run.smithi047.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-01-18T06:45:00.117 INFO:teuthology.orchestra.run.smithi047.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-01-18T06:45:00.117 INFO:teuthology.orchestra.run.smithi047.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-01-18T06:45:00.117 INFO:teuthology.orchestra.run.smithi047.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-01-18T06:45:00.117 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:45:00.117 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-01-18T06:45:00.117 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:45:00.117 INFO:teuthology.orchestra.run.smithi047.stdout:Install 6 Packages 2024-01-18T06:45:00.117 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:45:00.118 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 53 M 2024-01-18T06:45:00.118 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 229 M 2024-01-18T06:45:00.118 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-01-18T06:45:00.403 INFO:teuthology.orchestra.run.smithi047.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 1.8 MB/s | 169 kB 00:00 2024-01-18T06:45:00.428 INFO:teuthology.orchestra.run.smithi047.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 1.7 MB/s | 203 kB 00:00 2024-01-18T06:45:00.487 INFO:teuthology.orchestra.run.smithi047.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 5.4 MB/s | 323 kB 00:00 2024-01-18T06:45:00.512 INFO:teuthology.orchestra.run.smithi047.stdout:(4/6): oniguruma-6.8.2-3.el8.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-01-18T06:45:00.537 INFO:teuthology.orchestra.run.smithi047.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 71 kB 00:00 2024-01-18T06:45:01.146 INFO:teuthology.orchestra.run.smithi047.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 62 MB/s | 52 MB 00:00 2024-01-18T06:45:01.148 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-01-18T06:45:01.148 INFO:teuthology.orchestra.run.smithi047.stdout:Total 51 MB/s | 53 MB 00:01 2024-01-18T06:45:01.191 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-01-18T06:45:01.214 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-01-18T06:45:01.214 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-01-18T06:45:01.263 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-18T06:45:01.264 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:45:01.264 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-01-18T06:45:01.264 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:45:01.264 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-01-18T06:45:01.264 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-01-18T06:45:01.264 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-01-18T06:45:01.264 INFO:teuthology.orchestra.run.smithi194.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-01-18T06:45:01.264 INFO:teuthology.orchestra.run.smithi194.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-01-18T06:45:01.264 INFO:teuthology.orchestra.run.smithi194.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-01-18T06:45:01.264 INFO:teuthology.orchestra.run.smithi194.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-01-18T06:45:01.265 INFO:teuthology.orchestra.run.smithi194.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-01-18T06:45:01.265 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:45:01.265 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-01-18T06:45:01.265 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:45:01.265 INFO:teuthology.orchestra.run.smithi194.stdout:Install 6 Packages 2024-01-18T06:45:01.265 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:45:01.265 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 53 M 2024-01-18T06:45:01.265 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 229 M 2024-01-18T06:45:01.266 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-01-18T06:45:01.523 INFO:teuthology.orchestra.run.smithi194.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 2.8 MB/s | 169 kB 00:00 2024-01-18T06:45:01.540 INFO:teuthology.orchestra.run.smithi194.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 2.6 MB/s | 203 kB 00:00 2024-01-18T06:45:01.545 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-01-18T06:45:01.549 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-01-18T06:45:01.590 INFO:teuthology.orchestra.run.smithi194.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 6.3 MB/s | 323 kB 00:00 2024-01-18T06:45:01.615 INFO:teuthology.orchestra.run.smithi194.stdout:(4/6): oniguruma-6.8.2-3.el8.x86_64.rpm 2.0 MB/s | 188 kB 00:00 2024-01-18T06:45:01.632 INFO:teuthology.orchestra.run.smithi194.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.7 MB/s | 71 kB 00:00 2024-01-18T06:45:02.080 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-01-18T06:45:02.232 INFO:teuthology.orchestra.run.smithi194.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 67 MB/s | 52 MB 00:00 2024-01-18T06:45:02.234 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-01-18T06:45:02.235 INFO:teuthology.orchestra.run.smithi194.stdout:Total 55 MB/s | 53 MB 00:00 2024-01-18T06:45:02.279 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-01-18T06:45:02.297 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-01-18T06:45:02.297 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-01-18T06:45:02.330 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-01-18T06:45:02.536 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-01-18T06:45:02.614 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-01-18T06:45:02.617 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-01-18T06:45:02.691 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-18T06:45:02.801 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-18T06:45:02.949 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-01-18T06:45:03.095 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-01-18T06:45:03.149 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-18T06:45:03.298 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-01-18T06:45:03.482 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-01-18T06:45:03.646 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-18T06:45:03.728 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-18T06:45:03.903 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-01-18T06:45:04.046 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-18T06:45:08.364 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-18T06:45:08.622 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-18T06:45:08.883 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-18T06:45:08.932 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-18T06:45:08.932 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-01-18T06:45:08.932 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-01-18T06:45:08.932 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-01-18T06:45:08.933 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-01-18T06:45:08.933 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-01-18T06:45:09.127 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-18T06:45:09.362 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-01-18T06:45:09.362 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:45:09.362 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-01-18T06:45:09.362 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-01-18T06:45:09.362 INFO:teuthology.orchestra.run.smithi047.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-01-18T06:45:09.362 INFO:teuthology.orchestra.run.smithi047.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-01-18T06:45:09.362 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:45:09.362 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-01-18T06:45:09.399 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-18T06:45:09.399 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-01-18T06:45:09.399 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-01-18T06:45:09.399 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-01-18T06:45:09.399 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-01-18T06:45:09.399 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-01-18T06:45:09.611 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph 2024-01-18T06:45:09.811 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-01-18T06:45:09.811 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:45:09.811 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-01-18T06:45:09.811 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-01-18T06:45:09.811 INFO:teuthology.orchestra.run.smithi194.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-01-18T06:45:09.811 INFO:teuthology.orchestra.run.smithi194.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-01-18T06:45:09.811 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:45:09.811 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-18T06:45:10.218 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph 2024-01-18T06:45:10.366 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:00:56 ago on Thu 18 Jan 2024 06:44:14 AM UTC. 2024-01-18T06:45:10.950 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:00:52 ago on Thu 18 Jan 2024 06:44:18 AM UTC. 2024-01-18T06:45:11.970 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-01-18T06:45:11.973 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:45:11.973 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-01-18T06:45:11.973 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:45:11.974 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-01-18T06:45:11.974 INFO:teuthology.orchestra.run.smithi047.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-01-18T06:45:11.974 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-01-18T06:45:11.974 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-01-18T06:45:11.974 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-01-18T06:45:11.974 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-01-18T06:45:11.974 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-01-18T06:45:11.974 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-01-18T06:45:11.974 INFO:teuthology.orchestra.run.smithi047.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-18T06:45:11.974 INFO:teuthology.orchestra.run.smithi047.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-18T06:45:11.974 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-01-18T06:45:11.974 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-18T06:45:11.974 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-18T06:45:11.974 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-01-18T06:45:11.975 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-18T06:45:11.975 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-18T06:45:11.975 INFO:teuthology.orchestra.run.smithi047.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-18T06:45:11.975 INFO:teuthology.orchestra.run.smithi047.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-01-18T06:45:11.975 INFO:teuthology.orchestra.run.smithi047.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-01-18T06:45:11.975 INFO:teuthology.orchestra.run.smithi047.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-18T06:45:11.975 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-18T06:45:11.975 INFO:teuthology.orchestra.run.smithi047.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-18T06:45:11.975 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-01-18T06:45:11.975 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-01-18T06:45:11.975 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-01-18T06:45:11.975 INFO:teuthology.orchestra.run.smithi047.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-18T06:45:11.975 INFO:teuthology.orchestra.run.smithi047.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-18T06:45:11.975 INFO:teuthology.orchestra.run.smithi047.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-18T06:45:11.976 INFO:teuthology.orchestra.run.smithi047.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-18T06:45:11.976 INFO:teuthology.orchestra.run.smithi047.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-18T06:45:11.976 INFO:teuthology.orchestra.run.smithi047.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-18T06:45:11.976 INFO:teuthology.orchestra.run.smithi047.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-18T06:45:11.976 INFO:teuthology.orchestra.run.smithi047.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-01-18T06:45:11.976 INFO:teuthology.orchestra.run.smithi047.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-18T06:45:11.976 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:45:11.976 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-01-18T06:45:11.976 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:45:11.976 INFO:teuthology.orchestra.run.smithi047.stdout:Install 32 Packages 2024-01-18T06:45:11.976 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:45:11.977 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 31 M 2024-01-18T06:45:11.977 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 111 M 2024-01-18T06:45:11.977 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-01-18T06:45:12.303 INFO:teuthology.orchestra.run.smithi047.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 87 kB/s | 6.3 kB 00:00 2024-01-18T06:45:12.366 INFO:teuthology.orchestra.run.smithi047.stdout:(2/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 18 MB/s | 2.4 MB 00:00 2024-01-18T06:45:12.408 INFO:teuthology.orchestra.run.smithi047.stdout:(3/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 8.8 MB/s | 1.6 MB 00:00 2024-01-18T06:45:12.433 INFO:teuthology.orchestra.run.smithi047.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.7 MB/s | 220 kB 00:00 2024-01-18T06:45:12.446 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-18T06:45:12.449 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:45:12.449 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-01-18T06:45:12.449 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:45:12.449 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-01-18T06:45:12.450 INFO:teuthology.orchestra.run.smithi194.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-01-18T06:45:12.450 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-01-18T06:45:12.450 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-01-18T06:45:12.450 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-01-18T06:45:12.450 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-01-18T06:45:12.450 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-01-18T06:45:12.450 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-01-18T06:45:12.450 INFO:teuthology.orchestra.run.smithi194.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-18T06:45:12.450 INFO:teuthology.orchestra.run.smithi194.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-18T06:45:12.450 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-01-18T06:45:12.450 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-18T06:45:12.450 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-18T06:45:12.450 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-01-18T06:45:12.451 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-18T06:45:12.451 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-18T06:45:12.451 INFO:teuthology.orchestra.run.smithi194.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-18T06:45:12.451 INFO:teuthology.orchestra.run.smithi194.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-01-18T06:45:12.451 INFO:teuthology.orchestra.run.smithi194.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-01-18T06:45:12.451 INFO:teuthology.orchestra.run.smithi194.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-18T06:45:12.451 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-18T06:45:12.451 INFO:teuthology.orchestra.run.smithi194.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-18T06:45:12.451 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-01-18T06:45:12.451 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-01-18T06:45:12.451 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-01-18T06:45:12.451 INFO:teuthology.orchestra.run.smithi194.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-18T06:45:12.451 INFO:teuthology.orchestra.run.smithi194.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-18T06:45:12.451 INFO:teuthology.orchestra.run.smithi194.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-18T06:45:12.452 INFO:teuthology.orchestra.run.smithi194.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-18T06:45:12.452 INFO:teuthology.orchestra.run.smithi194.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-18T06:45:12.452 INFO:teuthology.orchestra.run.smithi194.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-18T06:45:12.452 INFO:teuthology.orchestra.run.smithi194.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-18T06:45:12.452 INFO:teuthology.orchestra.run.smithi194.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-01-18T06:45:12.452 INFO:teuthology.orchestra.run.smithi194.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-18T06:45:12.452 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:45:12.452 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-01-18T06:45:12.452 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:45:12.452 INFO:teuthology.orchestra.run.smithi194.stdout:Install 32 Packages 2024-01-18T06:45:12.452 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:45:12.453 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 31 M 2024-01-18T06:45:12.453 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 111 M 2024-01-18T06:45:12.453 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-01-18T06:45:12.525 INFO:teuthology.orchestra.run.smithi047.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 22 MB/s | 5.0 MB 00:00 2024-01-18T06:45:12.550 INFO:teuthology.orchestra.run.smithi047.stdout:(6/32): python3-mako-1.0.6-14.el8.noarch.rpm 1.3 MB/s | 157 kB 00:00 2024-01-18T06:45:12.600 INFO:teuthology.orchestra.run.smithi047.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 2.0 MB/s | 103 kB 00:00 2024-01-18T06:45:12.626 INFO:teuthology.orchestra.run.smithi047.stdout:(8/32): python3-markupsafe-0.23-19.el8.x86_64.r 386 kB/s | 39 kB 00:00 2024-01-18T06:45:12.651 INFO:teuthology.orchestra.run.smithi047.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 1.1 MB/s | 57 kB 00:00 2024-01-18T06:45:12.701 INFO:teuthology.orchestra.run.smithi047.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 5.9 MB/s | 457 kB 00:00 2024-01-18T06:45:12.760 INFO:teuthology.orchestra.run.smithi047.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 2.2 MB/s | 249 kB 00:00 2024-01-18T06:45:12.977 INFO:teuthology.orchestra.run.smithi047.stdout:(12/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 30 MB/s | 18 MB 00:00 2024-01-18T06:45:12.994 INFO:teuthology.orchestra.run.smithi047.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 467 kB/s | 109 kB 00:00 2024-01-18T06:45:13.027 INFO:teuthology.orchestra.run.smithi047.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 1.9 MB/s | 641 kB 00:00 2024-01-18T06:45:13.157 INFO:teuthology.orchestra.run.smithi194.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 136 kB/s | 6.3 kB 00:00 2024-01-18T06:45:13.207 INFO:teuthology.orchestra.run.smithi194.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-01-18T06:45:13.266 INFO:teuthology.orchestra.run.smithi194.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 15 MB/s | 2.4 MB 00:00 2024-01-18T06:45:13.292 INFO:teuthology.orchestra.run.smithi194.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.7 MB/s | 220 kB 00:00 2024-01-18T06:45:13.303 INFO:teuthology.orchestra.run.smithi047.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 134 kB/s | 44 kB 00:00 2024-01-18T06:45:13.375 INFO:teuthology.orchestra.run.smithi194.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 23 MB/s | 5.0 MB 00:00 2024-01-18T06:45:13.395 INFO:teuthology.orchestra.run.smithi047.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 462 kB/s | 185 kB 00:00 2024-01-18T06:45:13.400 INFO:teuthology.orchestra.run.smithi194.stdout:(6/32): python3-mako-1.0.6-14.el8.noarch.rpm 1.4 MB/s | 157 kB 00:00 2024-01-18T06:45:13.443 INFO:teuthology.orchestra.run.smithi194.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 2.4 MB/s | 103 kB 00:00 2024-01-18T06:45:13.458 INFO:teuthology.orchestra.run.smithi047.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 401 kB/s | 173 kB 00:00 2024-01-18T06:45:13.468 INFO:teuthology.orchestra.run.smithi194.stdout:(8/32): python3-markupsafe-0.23-19.el8.x86_64.r 421 kB/s | 39 kB 00:00 2024-01-18T06:45:13.493 INFO:teuthology.orchestra.run.smithi194.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 1.1 MB/s | 57 kB 00:00 2024-01-18T06:45:13.495 INFO:teuthology.orchestra.run.smithi047.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 111 kB/s | 11 kB 00:00 2024-01-18T06:45:13.543 INFO:teuthology.orchestra.run.smithi194.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 6.0 MB/s | 457 kB 00:00 2024-01-18T06:45:13.546 INFO:teuthology.orchestra.run.smithi047.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 202 kB/s | 18 kB 00:00 2024-01-18T06:45:13.571 INFO:teuthology.orchestra.run.smithi047.stdout:(20/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.4 MB/s | 384 kB 00:00 2024-01-18T06:45:13.577 INFO:teuthology.orchestra.run.smithi194.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 2.9 MB/s | 249 kB 00:00 2024-01-18T06:45:13.588 INFO:teuthology.orchestra.run.smithi047.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 528 kB/s | 49 kB 00:00 2024-01-18T06:45:13.638 INFO:teuthology.orchestra.run.smithi047.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 637 kB/s | 59 kB 00:00 2024-01-18T06:45:13.697 INFO:teuthology.orchestra.run.smithi047.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.2 MB/s | 283 kB 00:00 2024-01-18T06:45:13.745 INFO:teuthology.orchestra.run.smithi047.stdout:(24/32): python3-portend-2.6-1.el8.noarch.rpm 99 kB/s | 16 kB 00:00 2024-01-18T06:45:13.769 INFO:teuthology.orchestra.run.smithi194.stdout:(12/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 32 MB/s | 18 MB 00:00 2024-01-18T06:45:13.772 INFO:teuthology.orchestra.run.smithi047.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 140 kB/s | 19 kB 00:00 2024-01-18T06:45:13.794 INFO:teuthology.orchestra.run.smithi194.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 503 kB/s | 109 kB 00:00 2024-01-18T06:45:13.798 INFO:teuthology.orchestra.run.smithi047.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 238 kB/s | 24 kB 00:00 2024-01-18T06:45:13.848 INFO:teuthology.orchestra.run.smithi047.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 284 kB/s | 29 kB 00:00 2024-01-18T06:45:13.853 INFO:teuthology.orchestra.run.smithi194.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 2.0 MB/s | 641 kB 00:00 2024-01-18T06:45:13.873 INFO:teuthology.orchestra.run.smithi047.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 274 kB/s | 27 kB 00:00 2024-01-18T06:45:13.878 INFO:teuthology.orchestra.run.smithi194.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 402 kB/s | 44 kB 00:00 2024-01-18T06:45:13.899 INFO:teuthology.orchestra.run.smithi047.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.5 MB/s | 255 kB 00:00 2024-01-18T06:45:13.945 INFO:teuthology.orchestra.run.smithi194.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.2 MB/s | 185 kB 00:00 2024-01-18T06:45:13.974 INFO:teuthology.orchestra.run.smithi047.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 850 kB/s | 85 kB 00:00 2024-01-18T06:45:13.987 INFO:teuthology.orchestra.run.smithi194.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 267 kB/s | 11 kB 00:00 2024-01-18T06:45:14.029 INFO:teuthology.orchestra.run.smithi194.stdout:(18/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 982 kB/s | 173 kB 00:00 2024-01-18T06:45:14.030 INFO:teuthology.orchestra.run.smithi047.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 174 kB/s | 23 kB 00:00 2024-01-18T06:45:14.054 INFO:teuthology.orchestra.run.smithi194.stdout:(19/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.1 MB/s | 384 kB 00:00 2024-01-18T06:45:14.058 INFO:teuthology.orchestra.run.smithi047.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.2 MB/s | 251 kB 00:00 2024-01-18T06:45:14.060 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-01-18T06:45:14.060 INFO:teuthology.orchestra.run.smithi047.stdout:Total 15 MB/s | 31 MB 00:02 2024-01-18T06:45:14.079 INFO:teuthology.orchestra.run.smithi194.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 191 kB/s | 18 kB 00:00 2024-01-18T06:45:14.104 INFO:teuthology.orchestra.run.smithi194.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 646 kB/s | 49 kB 00:00 2024-01-18T06:45:14.131 INFO:teuthology.orchestra.run.smithi194.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 1.1 MB/s | 59 kB 00:00 2024-01-18T06:45:14.144 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-01-18T06:45:14.155 INFO:teuthology.orchestra.run.smithi194.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 309 kB/s | 16 kB 00:00 2024-01-18T06:45:14.181 INFO:teuthology.orchestra.run.smithi194.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 371 kB/s | 19 kB 00:00 2024-01-18T06:45:14.185 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-01-18T06:45:14.186 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-01-18T06:45:14.238 INFO:teuthology.orchestra.run.smithi194.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 288 kB/s | 24 kB 00:00 2024-01-18T06:45:14.264 INFO:teuthology.orchestra.run.smithi194.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 350 kB/s | 29 kB 00:00 2024-01-18T06:45:14.290 INFO:teuthology.orchestra.run.smithi194.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 541 kB/s | 27 kB 00:00 2024-01-18T06:45:14.315 INFO:teuthology.orchestra.run.smithi194.stdout:(28/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.2 MB/s | 283 kB 00:00 2024-01-18T06:45:14.390 INFO:teuthology.orchestra.run.smithi194.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.1 MB/s | 85 kB 00:00 2024-01-18T06:45:14.415 INFO:teuthology.orchestra.run.smithi194.stdout:(30/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.7 MB/s | 255 kB 00:00 2024-01-18T06:45:14.440 INFO:teuthology.orchestra.run.smithi194.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.6 MB/s | 251 kB 00:00 2024-01-18T06:45:14.465 INFO:teuthology.orchestra.run.smithi194.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 306 kB/s | 23 kB 00:00 2024-01-18T06:45:14.467 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-01-18T06:45:14.467 INFO:teuthology.orchestra.run.smithi194.stdout:Total 15 MB/s | 31 MB 00:02 2024-01-18T06:45:14.511 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-01-18T06:45:14.514 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-01-18T06:45:14.547 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-01-18T06:45:14.586 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-01-18T06:45:14.587 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-01-18T06:45:14.894 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-01-18T06:45:14.897 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-01-18T06:45:15.722 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-01-18T06:45:15.915 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-01-18T06:45:16.020 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-01-18T06:45:16.139 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-01-18T06:45:16.216 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-01-18T06:45:16.419 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-01-18T06:45:16.443 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-01-18T06:45:16.671 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-01-18T06:45:16.676 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-01-18T06:45:16.886 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-01-18T06:45:16.908 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-01-18T06:45:17.068 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-01-18T06:45:17.098 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-01-18T06:45:17.259 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-01-18T06:45:17.261 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-01-18T06:45:17.422 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-01-18T06:45:17.445 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-01-18T06:45:17.583 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-01-18T06:45:17.641 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-01-18T06:45:17.816 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-01-18T06:45:17.885 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-01-18T06:45:18.037 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-01-18T06:45:18.123 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-01-18T06:45:18.242 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-01-18T06:45:18.390 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-01-18T06:45:18.473 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-01-18T06:45:18.786 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-01-18T06:45:18.857 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-01-18T06:45:19.034 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-01-18T06:45:19.112 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-01-18T06:45:19.222 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-01-18T06:45:19.301 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-01-18T06:45:19.492 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-01-18T06:45:19.567 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-01-18T06:45:19.691 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-18T06:45:19.744 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-18T06:45:20.007 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-01-18T06:45:20.060 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-01-18T06:45:20.195 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-01-18T06:45:20.273 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-01-18T06:45:20.384 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-01-18T06:45:20.454 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-01-18T06:45:20.604 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-01-18T06:45:20.677 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-01-18T06:45:20.891 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-01-18T06:45:20.927 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-01-18T06:45:21.148 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-01-18T06:45:21.160 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-01-18T06:45:21.407 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-01-18T06:45:21.413 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-01-18T06:45:21.763 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-01-18T06:45:21.768 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-01-18T06:45:22.092 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-01-18T06:45:22.145 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-01-18T06:45:22.489 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-01-18T06:45:22.565 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-01-18T06:45:22.718 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-18T06:45:22.817 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-18T06:45:24.244 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-18T06:45:24.426 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-18T06:45:24.450 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-18T06:45:24.634 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-18T06:45:24.860 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-18T06:45:25.059 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-18T06:45:25.093 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-18T06:45:25.281 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-18T06:45:25.309 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-18T06:45:25.526 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-18T06:45:25.544 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-18T06:45:25.544 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-18T06:45:25.684 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-18T06:45:25.730 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-18T06:45:25.748 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-18T06:45:25.905 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-18T06:45:26.082 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-18T06:45:26.082 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-01-18T06:45:26.082 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-01-18T06:45:26.082 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-01-18T06:45:26.082 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-01-18T06:45:26.082 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-01-18T06:45:26.082 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-01-18T06:45:26.082 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-01-18T06:45:26.082 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-01-18T06:45:26.082 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-01-18T06:45:26.083 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-01-18T06:45:26.083 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-01-18T06:45:26.083 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-01-18T06:45:26.083 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-01-18T06:45:26.083 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-01-18T06:45:26.083 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-01-18T06:45:26.083 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-01-18T06:45:26.083 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-18T06:45:26.083 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-01-18T06:45:26.083 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-01-18T06:45:26.083 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-01-18T06:45:26.083 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-01-18T06:45:26.083 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-01-18T06:45:26.083 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-01-18T06:45:26.083 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-01-18T06:45:26.084 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-01-18T06:45:26.084 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-01-18T06:45:26.084 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-01-18T06:45:26.084 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-01-18T06:45:26.084 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-01-18T06:45:26.084 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-01-18T06:45:26.084 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-01-18T06:45:26.333 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-18T06:45:26.334 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-01-18T06:45:26.334 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-01-18T06:45:26.334 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-01-18T06:45:26.334 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-01-18T06:45:26.334 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-01-18T06:45:26.334 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-01-18T06:45:26.334 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-01-18T06:45:26.334 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-01-18T06:45:26.334 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-01-18T06:45:26.334 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-01-18T06:45:26.334 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-01-18T06:45:26.334 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-01-18T06:45:26.335 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-01-18T06:45:26.335 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-01-18T06:45:26.335 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-01-18T06:45:26.335 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-01-18T06:45:26.335 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-18T06:45:26.335 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-01-18T06:45:26.335 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-01-18T06:45:26.335 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-01-18T06:45:26.335 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-01-18T06:45:26.335 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-01-18T06:45:26.335 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-01-18T06:45:26.335 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-01-18T06:45:26.335 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-01-18T06:45:26.335 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-01-18T06:45:26.335 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-01-18T06:45:26.336 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-01-18T06:45:26.336 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-01-18T06:45:26.336 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-01-18T06:45:26.336 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-01-18T06:45:27.030 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-01-18T06:45:27.030 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:45:27.030 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-01-18T06:45:27.030 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-01-18T06:45:27.030 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-01-18T06:45:27.030 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-01-18T06:45:27.030 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-01-18T06:45:27.030 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-01-18T06:45:27.031 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-01-18T06:45:27.031 INFO:teuthology.orchestra.run.smithi194.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-18T06:45:27.031 INFO:teuthology.orchestra.run.smithi194.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-18T06:45:27.031 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-18T06:45:27.031 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-18T06:45:27.031 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-18T06:45:27.031 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-01-18T06:45:27.031 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-18T06:45:27.031 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-18T06:45:27.031 INFO:teuthology.orchestra.run.smithi194.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-18T06:45:27.031 INFO:teuthology.orchestra.run.smithi194.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-18T06:45:27.031 INFO:teuthology.orchestra.run.smithi194.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-18T06:45:27.031 INFO:teuthology.orchestra.run.smithi194.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-18T06:45:27.031 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-18T06:45:27.031 INFO:teuthology.orchestra.run.smithi194.stdout: python3-portend-2.6-1.el8.noarch 2024-01-18T06:45:27.032 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-18T06:45:27.032 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-18T06:45:27.032 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-18T06:45:27.032 INFO:teuthology.orchestra.run.smithi194.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-18T06:45:27.032 INFO:teuthology.orchestra.run.smithi194.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-18T06:45:27.032 INFO:teuthology.orchestra.run.smithi194.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-18T06:45:27.032 INFO:teuthology.orchestra.run.smithi194.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-18T06:45:27.032 INFO:teuthology.orchestra.run.smithi194.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-18T06:45:27.032 INFO:teuthology.orchestra.run.smithi194.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-18T06:45:27.032 INFO:teuthology.orchestra.run.smithi194.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-18T06:45:27.032 INFO:teuthology.orchestra.run.smithi194.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-18T06:45:27.032 INFO:teuthology.orchestra.run.smithi194.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-18T06:45:27.032 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:45:27.032 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-18T06:45:27.253 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-01-18T06:45:27.253 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:45:27.253 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-01-18T06:45:27.253 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-01-18T06:45:27.254 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-01-18T06:45:27.254 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-01-18T06:45:27.254 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-01-18T06:45:27.254 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-01-18T06:45:27.254 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-01-18T06:45:27.254 INFO:teuthology.orchestra.run.smithi047.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-18T06:45:27.254 INFO:teuthology.orchestra.run.smithi047.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-18T06:45:27.254 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-18T06:45:27.254 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-18T06:45:27.254 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-18T06:45:27.254 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-01-18T06:45:27.254 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-18T06:45:27.254 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-18T06:45:27.254 INFO:teuthology.orchestra.run.smithi047.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-18T06:45:27.255 INFO:teuthology.orchestra.run.smithi047.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-18T06:45:27.255 INFO:teuthology.orchestra.run.smithi047.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-18T06:45:27.255 INFO:teuthology.orchestra.run.smithi047.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-18T06:45:27.255 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-18T06:45:27.255 INFO:teuthology.orchestra.run.smithi047.stdout: python3-portend-2.6-1.el8.noarch 2024-01-18T06:45:27.255 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-18T06:45:27.255 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-18T06:45:27.255 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-18T06:45:27.255 INFO:teuthology.orchestra.run.smithi047.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-18T06:45:27.255 INFO:teuthology.orchestra.run.smithi047.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-18T06:45:27.255 INFO:teuthology.orchestra.run.smithi047.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-18T06:45:27.255 INFO:teuthology.orchestra.run.smithi047.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-18T06:45:27.255 INFO:teuthology.orchestra.run.smithi047.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-18T06:45:27.255 INFO:teuthology.orchestra.run.smithi047.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-18T06:45:27.256 INFO:teuthology.orchestra.run.smithi047.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-18T06:45:27.256 INFO:teuthology.orchestra.run.smithi047.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-18T06:45:27.256 INFO:teuthology.orchestra.run.smithi047.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-18T06:45:27.256 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:45:27.256 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-01-18T06:45:27.271 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-base 2024-01-18T06:45:27.517 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-base 2024-01-18T06:45:28.003 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:10 ago on Thu 18 Jan 2024 06:44:18 AM UTC. 2024-01-18T06:45:28.276 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:14 ago on Thu 18 Jan 2024 06:44:14 AM UTC. 2024-01-18T06:45:29.442 INFO:teuthology.orchestra.run.smithi194.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-18T06:45:29.488 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-18T06:45:29.489 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-01-18T06:45:29.489 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-18T06:45:29.590 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install cephadm 2024-01-18T06:45:29.825 INFO:teuthology.orchestra.run.smithi047.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-18T06:45:29.875 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-01-18T06:45:29.876 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-01-18T06:45:29.876 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-01-18T06:45:29.981 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install cephadm 2024-01-18T06:45:30.310 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:12 ago on Thu 18 Jan 2024 06:44:18 AM UTC. 2024-01-18T06:45:30.741 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:16 ago on Thu 18 Jan 2024 06:44:14 AM UTC. 2024-01-18T06:45:31.794 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-18T06:45:31.795 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:45:31.795 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-01-18T06:45:31.795 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:45:31.795 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-01-18T06:45:31.795 INFO:teuthology.orchestra.run.smithi194.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-01-18T06:45:31.795 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:45:31.795 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-01-18T06:45:31.795 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:45:31.795 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-01-18T06:45:31.795 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:45:31.795 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 71 k 2024-01-18T06:45:31.795 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 290 k 2024-01-18T06:45:31.795 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-01-18T06:45:31.851 INFO:teuthology.orchestra.run.smithi194.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.3 MB/s | 71 kB 00:00 2024-01-18T06:45:31.851 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-01-18T06:45:31.852 INFO:teuthology.orchestra.run.smithi194.stdout:Total 1.2 MB/s | 71 kB 00:00 2024-01-18T06:45:31.852 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-01-18T06:45:31.859 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-01-18T06:45:31.859 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-01-18T06:45:31.860 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-01-18T06:45:31.863 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-01-18T06:45:31.903 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-01-18T06:45:32.324 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-01-18T06:45:32.324 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:45:32.324 INFO:teuthology.orchestra.run.smithi047.stdout: Package Architecture Version Repository Size 2024-01-18T06:45:32.324 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:45:32.325 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-01-18T06:45:32.325 INFO:teuthology.orchestra.run.smithi047.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-01-18T06:45:32.325 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:45:32.325 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-01-18T06:45:32.325 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:45:32.325 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-01-18T06:45:32.325 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:45:32.327 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 71 k 2024-01-18T06:45:32.327 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 290 k 2024-01-18T06:45:32.327 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-01-18T06:45:32.379 INFO:teuthology.orchestra.run.smithi047.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.3 MB/s | 71 kB 00:00 2024-01-18T06:45:32.380 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-01-18T06:45:32.380 INFO:teuthology.orchestra.run.smithi047.stdout:Total 1.3 MB/s | 71 kB 00:00 2024-01-18T06:45:32.380 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-01-18T06:45:32.381 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-18T06:45:32.387 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-01-18T06:45:32.387 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-01-18T06:45:32.389 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-01-18T06:45:32.393 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-01-18T06:45:32.439 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-01-18T06:45:32.579 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-18T06:45:32.806 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-18T06:45:33.001 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-18T06:45:33.156 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-18T06:45:33.156 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:45:33.156 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-01-18T06:45:33.156 INFO:teuthology.orchestra.run.smithi194.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-01-18T06:45:33.156 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:45:33.156 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-18T06:45:33.213 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-18T06:45:33.412 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-immutable-object-cache 2024-01-18T06:45:33.454 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-18T06:45:33.839 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-18T06:45:33.839 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:45:33.839 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-01-18T06:45:33.839 INFO:teuthology.orchestra.run.smithi047.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-01-18T06:45:33.840 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:45:33.840 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-01-18T06:45:34.091 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-immutable-object-cache 2024-01-18T06:45:34.146 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:16 ago on Thu 18 Jan 2024 06:44:18 AM UTC. 2024-01-18T06:45:34.857 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:20 ago on Thu 18 Jan 2024 06:44:14 AM UTC. 2024-01-18T06:45:35.610 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-18T06:45:35.610 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:45:35.610 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-01-18T06:45:35.610 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:45:35.610 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-01-18T06:45:35.610 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-01-18T06:45:35.611 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:45:35.611 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-01-18T06:45:35.611 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:45:35.611 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-01-18T06:45:35.611 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:45:35.611 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 163 k 2024-01-18T06:45:35.611 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 587 k 2024-01-18T06:45:35.611 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-01-18T06:45:35.654 INFO:teuthology.orchestra.run.smithi194.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.7 MB/s | 163 kB 00:00 2024-01-18T06:45:35.655 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-01-18T06:45:35.655 INFO:teuthology.orchestra.run.smithi194.stdout:Total 3.6 MB/s | 163 kB 00:00 2024-01-18T06:45:35.655 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-01-18T06:45:35.666 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-01-18T06:45:35.666 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-01-18T06:45:35.714 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-01-18T06:45:35.716 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-01-18T06:45:35.834 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-01-18T06:45:35.999 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T06:45:36.448 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-01-18T06:45:36.449 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:45:36.449 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-01-18T06:45:36.449 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:45:36.449 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-01-18T06:45:36.449 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-01-18T06:45:36.449 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:45:36.449 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-01-18T06:45:36.450 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:45:36.450 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-01-18T06:45:36.450 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:45:36.450 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 163 k 2024-01-18T06:45:36.451 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 587 k 2024-01-18T06:45:36.451 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-01-18T06:45:36.464 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T06:45:36.518 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 2.3 MB/s | 163 kB 00:00 2024-01-18T06:45:36.518 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-01-18T06:45:36.519 INFO:teuthology.orchestra.run.smithi047.stdout:Total 2.3 MB/s | 163 kB 00:00 2024-01-18T06:45:36.519 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-01-18T06:45:36.531 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-01-18T06:45:36.531 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-01-18T06:45:36.581 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-01-18T06:45:36.584 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-01-18T06:45:36.700 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-01-18T06:45:36.895 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T06:45:37.312 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T06:45:37.723 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T06:45:37.724 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:45:37.724 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-01-18T06:45:37.724 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-01-18T06:45:37.724 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:45:37.725 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-01-18T06:45:37.871 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T06:45:37.871 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:45:37.871 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-01-18T06:45:37.871 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-01-18T06:45:37.871 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:45:37.871 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-18T06:45:37.950 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-mgr 2024-01-18T06:45:38.088 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr 2024-01-18T06:45:38.705 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:24 ago on Thu 18 Jan 2024 06:44:14 AM UTC. 2024-01-18T06:45:38.824 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:20 ago on Thu 18 Jan 2024 06:44:18 AM UTC. 2024-01-18T06:45:40.252 INFO:teuthology.orchestra.run.smithi047.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-18T06:45:40.296 INFO:teuthology.orchestra.run.smithi194.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-18T06:45:40.303 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-01-18T06:45:40.305 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-01-18T06:45:40.305 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-01-18T06:45:40.343 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-18T06:45:40.344 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-01-18T06:45:40.344 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-18T06:45:40.438 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr-dashboard 2024-01-18T06:45:40.964 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-mgr-dashboard 2024-01-18T06:45:41.161 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:23 ago on Thu 18 Jan 2024 06:44:18 AM UTC. 2024-01-18T06:45:41.721 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:27 ago on Thu 18 Jan 2024 06:44:14 AM UTC. 2024-01-18T06:45:42.644 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-18T06:45:42.645 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:45:42.645 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-01-18T06:45:42.645 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:45:42.645 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-01-18T06:45:42.645 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-01-18T06:45:42.645 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-01-18T06:45:42.645 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-01-18T06:45:42.645 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-01-18T06:45:42.645 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-01-18T06:45:42.645 INFO:teuthology.orchestra.run.smithi194.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-18T06:45:42.645 INFO:teuthology.orchestra.run.smithi194.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-18T06:45:42.645 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:45:42.646 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-01-18T06:45:42.646 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:45:42.646 INFO:teuthology.orchestra.run.smithi194.stdout:Install 6 Packages 2024-01-18T06:45:42.646 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:45:42.646 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 3.0 M 2024-01-18T06:45:42.646 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 15 M 2024-01-18T06:45:42.646 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-01-18T06:45:43.183 INFO:teuthology.orchestra.run.smithi194.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 67 kB/s | 24 kB 00:00 2024-01-18T06:45:43.266 INFO:teuthology.orchestra.run.smithi194.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 20 kB/s | 8.9 kB 00:00 2024-01-18T06:45:43.306 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-01-18T06:45:43.308 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:45:43.308 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-01-18T06:45:43.308 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:45:43.308 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-01-18T06:45:43.308 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-01-18T06:45:43.308 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-01-18T06:45:43.308 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-01-18T06:45:43.308 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-01-18T06:45:43.308 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-01-18T06:45:43.308 INFO:teuthology.orchestra.run.smithi047.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-18T06:45:43.308 INFO:teuthology.orchestra.run.smithi047.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-18T06:45:43.308 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:45:43.309 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-01-18T06:45:43.309 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:45:43.309 INFO:teuthology.orchestra.run.smithi047.stdout:Install 6 Packages 2024-01-18T06:45:43.309 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:45:43.309 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 3.0 M 2024-01-18T06:45:43.309 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 15 M 2024-01-18T06:45:43.309 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-01-18T06:45:43.325 INFO:teuthology.orchestra.run.smithi194.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 5.3 MB/s | 2.7 MB 00:00 2024-01-18T06:45:43.350 INFO:teuthology.orchestra.run.smithi194.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 255 kB/s | 43 kB 00:00 2024-01-18T06:45:43.438 INFO:teuthology.orchestra.run.smithi047.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 687 kB/s | 24 kB 00:00 2024-01-18T06:45:43.497 INFO:teuthology.orchestra.run.smithi047.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 95 kB/s | 8.9 kB 00:00 2024-01-18T06:45:43.556 INFO:teuthology.orchestra.run.smithi047.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 17 MB/s | 2.7 MB 00:00 2024-01-18T06:45:43.581 INFO:teuthology.orchestra.run.smithi047.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 299 kB/s | 43 kB 00:00 2024-01-18T06:45:43.615 INFO:teuthology.orchestra.run.smithi047.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 287 kB/s | 34 kB 00:00 2024-01-18T06:45:43.673 INFO:teuthology.orchestra.run.smithi047.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.6 MB/s | 196 kB 00:00 2024-01-18T06:45:43.675 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-01-18T06:45:43.675 INFO:teuthology.orchestra.run.smithi047.stdout:Total 8.1 MB/s | 3.0 MB 00:00 2024-01-18T06:45:43.685 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-01-18T06:45:43.696 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-01-18T06:45:43.696 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-01-18T06:45:43.709 INFO:teuthology.orchestra.run.smithi194.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 76 kB/s | 34 kB 00:00 2024-01-18T06:45:43.888 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-01-18T06:45:43.891 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-01-18T06:45:43.975 INFO:teuthology.orchestra.run.smithi194.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 302 kB/s | 196 kB 00:00 2024-01-18T06:45:43.979 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-01-18T06:45:43.979 INFO:teuthology.orchestra.run.smithi194.stdout:Total 2.2 MB/s | 3.0 MB 00:01 2024-01-18T06:45:43.991 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-01-18T06:45:44.001 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-01-18T06:45:44.001 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-01-18T06:45:44.175 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-01-18T06:45:44.178 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-01-18T06:45:44.195 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-01-18T06:45:44.436 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-01-18T06:45:44.444 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-01-18T06:45:44.628 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-01-18T06:45:44.738 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-01-18T06:45:44.865 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-01-18T06:45:44.954 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-01-18T06:45:45.058 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-01-18T06:45:45.060 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-01-18T06:45:45.179 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-01-18T06:45:46.127 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-01-18T06:45:46.163 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-01-18T06:45:46.422 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-18T06:45:46.690 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-18T06:45:46.690 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-01-18T06:45:46.690 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-01-18T06:45:46.690 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-01-18T06:45:46.690 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-01-18T06:45:46.690 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-01-18T06:45:46.755 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-18T06:45:47.104 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-18T06:45:47.104 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-01-18T06:45:47.104 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-01-18T06:45:47.104 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-01-18T06:45:47.104 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-01-18T06:45:47.104 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-01-18T06:45:47.106 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-01-18T06:45:47.106 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:45:47.106 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-01-18T06:45:47.106 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-01-18T06:45:47.106 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-01-18T06:45:47.106 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-01-18T06:45:47.107 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-18T06:45:47.107 INFO:teuthology.orchestra.run.smithi194.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-18T06:45:47.107 INFO:teuthology.orchestra.run.smithi194.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-18T06:45:47.107 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:45:47.107 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-18T06:45:47.219 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-18T06:45:47.525 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-01-18T06:45:47.525 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:45:47.525 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-01-18T06:45:47.526 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-01-18T06:45:47.526 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-01-18T06:45:47.526 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-01-18T06:45:47.526 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-18T06:45:47.526 INFO:teuthology.orchestra.run.smithi047.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-18T06:45:47.526 INFO:teuthology.orchestra.run.smithi047.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-18T06:45:47.526 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:45:47.526 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-01-18T06:45:47.643 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-18T06:45:47.938 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:29 ago on Thu 18 Jan 2024 06:44:18 AM UTC. 2024-01-18T06:45:48.403 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:34 ago on Thu 18 Jan 2024 06:44:14 AM UTC. 2024-01-18T06:45:49.428 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-18T06:45:49.429 INFO:teuthology.orchestra.run.smithi194.stdout:============================================================================================ 2024-01-18T06:45:49.429 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-01-18T06:45:49.429 INFO:teuthology.orchestra.run.smithi194.stdout:============================================================================================ 2024-01-18T06:45:49.429 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-01-18T06:45:49.429 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-01-18T06:45:49.429 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-01-18T06:45:49.429 INFO:teuthology.orchestra.run.smithi194.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-01-18T06:45:49.429 INFO:teuthology.orchestra.run.smithi194.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-01-18T06:45:49.429 INFO:teuthology.orchestra.run.smithi194.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-01-18T06:45:49.429 INFO:teuthology.orchestra.run.smithi194.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-01-18T06:45:49.429 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-01-18T06:45:49.430 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-01-18T06:45:49.430 INFO:teuthology.orchestra.run.smithi194.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-01-18T06:45:49.430 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:45:49.430 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-01-18T06:45:49.430 INFO:teuthology.orchestra.run.smithi194.stdout:============================================================================================ 2024-01-18T06:45:49.430 INFO:teuthology.orchestra.run.smithi194.stdout:Install 8 Packages 2024-01-18T06:45:49.430 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:45:49.431 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 37 M 2024-01-18T06:45:49.431 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 210 M 2024-01-18T06:45:49.431 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-01-18T06:45:49.805 INFO:teuthology.orchestra.run.smithi194.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 57 MB/s | 8.6 MB 00:00 2024-01-18T06:45:49.938 INFO:teuthology.orchestra.run.smithi194.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 28 MB/s | 3.7 MB 00:00 2024-01-18T06:45:49.972 INFO:teuthology.orchestra.run.smithi194.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 6.6 MB/s | 225 kB 00:00 2024-01-18T06:45:49.992 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-01-18T06:45:49.994 INFO:teuthology.orchestra.run.smithi047.stdout:============================================================================================ 2024-01-18T06:45:49.994 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-01-18T06:45:49.994 INFO:teuthology.orchestra.run.smithi047.stdout:============================================================================================ 2024-01-18T06:45:49.994 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-01-18T06:45:49.994 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-01-18T06:45:49.994 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-01-18T06:45:49.994 INFO:teuthology.orchestra.run.smithi047.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-01-18T06:45:49.994 INFO:teuthology.orchestra.run.smithi047.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-01-18T06:45:49.994 INFO:teuthology.orchestra.run.smithi047.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-01-18T06:45:49.994 INFO:teuthology.orchestra.run.smithi047.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-01-18T06:45:49.994 INFO:teuthology.orchestra.run.smithi047.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-01-18T06:45:49.994 INFO:teuthology.orchestra.run.smithi047.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-01-18T06:45:49.995 INFO:teuthology.orchestra.run.smithi047.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-01-18T06:45:49.995 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:45:49.995 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-01-18T06:45:49.995 INFO:teuthology.orchestra.run.smithi047.stdout:============================================================================================ 2024-01-18T06:45:49.995 INFO:teuthology.orchestra.run.smithi047.stdout:Install 8 Packages 2024-01-18T06:45:49.995 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:45:49.995 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 37 M 2024-01-18T06:45:49.996 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 210 M 2024-01-18T06:45:49.996 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-01-18T06:45:50.055 INFO:teuthology.orchestra.run.smithi194.stdout:(4/8): openblas-0.3.15-4.el8.x86_64.rpm 12 MB/s | 4.7 MB 00:00 2024-01-18T06:45:50.131 INFO:teuthology.orchestra.run.smithi194.stdout:(5/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 10 MB/s | 4.9 MB 00:00 2024-01-18T06:45:50.323 INFO:teuthology.orchestra.run.smithi194.stdout:(6/8): python3-scipy-1.0.0-21.module_el8+762+77 40 MB/s | 14 MB 00:00 2024-01-18T06:45:50.348 INFO:teuthology.orchestra.run.smithi194.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 795 kB/s | 172 kB 00:00 2024-01-18T06:45:50.390 INFO:teuthology.orchestra.run.smithi194.stdout:(8/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.9 MB/s | 645 kB 00:00 2024-01-18T06:45:50.391 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-01-18T06:45:50.391 INFO:teuthology.orchestra.run.smithi194.stdout:Total 39 MB/s | 37 MB 00:00 2024-01-18T06:45:50.436 INFO:teuthology.orchestra.run.smithi047.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 41 MB/s | 8.6 MB 00:00 2024-01-18T06:45:50.578 INFO:teuthology.orchestra.run.smithi047.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 14 MB/s | 4.9 MB 00:00 2024-01-18T06:45:50.621 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-01-18T06:45:50.639 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-01-18T06:45:50.639 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-01-18T06:45:50.676 INFO:teuthology.orchestra.run.smithi047.stdout:(3/8): openblas-0.3.15-4.el8.x86_64.rpm 11 MB/s | 4.7 MB 00:00 2024-01-18T06:45:50.712 INFO:teuthology.orchestra.run.smithi047.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.6 MB/s | 225 kB 00:00 2024-01-18T06:45:50.787 INFO:teuthology.orchestra.run.smithi047.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 11 MB/s | 3.7 MB 00:00 2024-01-18T06:45:50.879 INFO:teuthology.orchestra.run.smithi047.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 1.8 MB/s | 172 kB 00:00 2024-01-18T06:45:50.904 INFO:teuthology.orchestra.run.smithi047.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 3.3 MB/s | 645 kB 00:00 2024-01-18T06:45:50.952 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-01-18T06:45:50.955 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-01-18T06:45:51.080 INFO:teuthology.orchestra.run.smithi047.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 35 MB/s | 14 MB 00:00 2024-01-18T06:45:51.081 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-01-18T06:45:51.081 INFO:teuthology.orchestra.run.smithi047.stdout:Total 34 MB/s | 37 MB 00:01 2024-01-18T06:45:51.317 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-01-18T06:45:51.337 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-01-18T06:45:51.337 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-01-18T06:45:51.563 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-01-18T06:45:51.684 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-01-18T06:45:51.687 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-01-18T06:45:51.705 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-18T06:45:51.839 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-18T06:45:51.979 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-18T06:45:52.395 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-01-18T06:45:52.518 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-18T06:45:52.564 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-18T06:45:52.660 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-18T06:45:52.738 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-01-18T06:45:52.776 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-18T06:45:53.347 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-01-18T06:45:53.415 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-18T06:45:53.563 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-18T06:45:53.617 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-01-18T06:45:54.269 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-01-18T06:45:54.438 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-18T06:45:54.442 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-18T06:45:54.774 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-01-18T06:45:54.971 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-18T06:45:55.179 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-18T06:45:55.525 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-01-18T06:45:55.767 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-18T06:45:56.727 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-18T06:45:57.639 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-18T06:45:58.011 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-01-18T06:45:58.228 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-18T06:45:58.543 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-18T06:45:58.544 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-01-18T06:45:58.544 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-01-18T06:45:58.544 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-01-18T06:45:58.544 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-01-18T06:45:58.544 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-01-18T06:45:58.544 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-01-18T06:45:58.544 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-01-18T06:45:58.947 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-01-18T06:45:58.948 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:45:58.948 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-01-18T06:45:58.948 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-01-18T06:45:58.948 INFO:teuthology.orchestra.run.smithi194.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-01-18T06:45:58.948 INFO:teuthology.orchestra.run.smithi194.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-01-18T06:45:58.948 INFO:teuthology.orchestra.run.smithi194.stdout: openblas-0.3.15-4.el8.x86_64 2024-01-18T06:45:58.948 INFO:teuthology.orchestra.run.smithi194.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-01-18T06:45:58.948 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-18T06:45:58.948 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-18T06:45:58.948 INFO:teuthology.orchestra.run.smithi194.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-01-18T06:45:58.948 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:45:58.948 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-18T06:45:59.056 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr-rook 2024-01-18T06:45:59.689 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-01-18T06:45:59.784 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:41 ago on Thu 18 Jan 2024 06:44:18 AM UTC. 2024-01-18T06:45:59.903 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-18T06:46:00.251 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-18T06:46:00.251 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-01-18T06:46:00.251 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-01-18T06:46:00.252 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-01-18T06:46:00.252 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-01-18T06:46:00.252 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-01-18T06:46:00.252 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-01-18T06:46:00.252 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-01-18T06:46:00.702 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-01-18T06:46:00.702 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:46:00.702 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-01-18T06:46:00.702 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-01-18T06:46:00.702 INFO:teuthology.orchestra.run.smithi047.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-01-18T06:46:00.702 INFO:teuthology.orchestra.run.smithi047.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-01-18T06:46:00.702 INFO:teuthology.orchestra.run.smithi047.stdout: openblas-0.3.15-4.el8.x86_64 2024-01-18T06:46:00.702 INFO:teuthology.orchestra.run.smithi047.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-01-18T06:46:00.702 INFO:teuthology.orchestra.run.smithi047.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-18T06:46:00.702 INFO:teuthology.orchestra.run.smithi047.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-18T06:46:00.703 INFO:teuthology.orchestra.run.smithi047.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-01-18T06:46:00.703 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:46:00.703 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-01-18T06:46:00.850 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-mgr-rook 2024-01-18T06:46:01.264 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-18T06:46:01.266 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:46:01.266 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-01-18T06:46:01.266 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:46:01.266 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-01-18T06:46:01.266 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-01-18T06:46:01.266 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-01-18T06:46:01.266 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-18T06:46:01.266 INFO:teuthology.orchestra.run.smithi194.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-18T06:46:01.266 INFO:teuthology.orchestra.run.smithi194.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-18T06:46:01.267 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-01-18T06:46:01.267 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-01-18T06:46:01.267 INFO:teuthology.orchestra.run.smithi194.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-18T06:46:01.267 INFO:teuthology.orchestra.run.smithi194.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-01-18T06:46:01.267 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-01-18T06:46:01.267 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-01-18T06:46:01.267 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-01-18T06:46:01.267 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-18T06:46:01.267 INFO:teuthology.orchestra.run.smithi194.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-18T06:46:01.267 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:46:01.267 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-01-18T06:46:01.267 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:46:01.267 INFO:teuthology.orchestra.run.smithi194.stdout:Install 13 Packages 2024-01-18T06:46:01.267 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:46:01.268 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 1.9 M 2024-01-18T06:46:01.268 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 24 M 2024-01-18T06:46:01.268 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-01-18T06:46:01.589 INFO:teuthology.orchestra.run.smithi194.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 622 kB/s | 27 kB 00:00 2024-01-18T06:46:01.605 INFO:teuthology.orchestra.run.smithi194.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 443 kB/s | 27 kB 00:00 2024-01-18T06:46:01.606 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:47 ago on Thu 18 Jan 2024 06:44:14 AM UTC. 2024-01-18T06:46:01.622 INFO:teuthology.orchestra.run.smithi194.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 260 kB/s | 20 kB 00:00 2024-01-18T06:46:01.673 INFO:teuthology.orchestra.run.smithi194.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 2.1 MB/s | 110 kB 00:00 2024-01-18T06:46:01.698 INFO:teuthology.orchestra.run.smithi194.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.1 MB/s | 126 kB 00:00 2024-01-18T06:46:01.723 INFO:teuthology.orchestra.run.smithi194.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 1.5 MB/s | 155 kB 00:00 2024-01-18T06:46:01.748 INFO:teuthology.orchestra.run.smithi194.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 565 kB/s | 43 kB 00:00 2024-01-18T06:46:01.999 INFO:teuthology.orchestra.run.smithi194.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 59 kB/s | 16 kB 00:00 2024-01-18T06:46:02.024 INFO:teuthology.orchestra.run.smithi194.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 101 kB/s | 33 kB 00:00 2024-01-18T06:46:02.116 INFO:teuthology.orchestra.run.smithi194.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 221 kB/s | 81 kB 00:00 2024-01-18T06:46:02.208 INFO:teuthology.orchestra.run.smithi194.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 350 kB/s | 64 kB 00:00 2024-01-18T06:46:02.250 INFO:teuthology.orchestra.run.smithi194.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 459 kB/s | 61 kB 00:00 2024-01-18T06:46:02.633 INFO:teuthology.orchestra.run.smithi194.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 1.8 MB/s | 1.2 MB 00:00 2024-01-18T06:46:02.633 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-01-18T06:46:02.634 INFO:teuthology.orchestra.run.smithi194.stdout:Total 1.4 MB/s | 1.9 MB 00:01 2024-01-18T06:46:02.689 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-01-18T06:46:02.705 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-01-18T06:46:02.705 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-01-18T06:46:02.793 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-01-18T06:46:02.800 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-01-18T06:46:03.215 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-01-18T06:46:03.217 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:46:03.217 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-01-18T06:46:03.218 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:46:03.218 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-01-18T06:46:03.218 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-01-18T06:46:03.218 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-01-18T06:46:03.218 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-18T06:46:03.218 INFO:teuthology.orchestra.run.smithi047.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-18T06:46:03.218 INFO:teuthology.orchestra.run.smithi047.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-18T06:46:03.218 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-01-18T06:46:03.218 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-01-18T06:46:03.218 INFO:teuthology.orchestra.run.smithi047.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-18T06:46:03.218 INFO:teuthology.orchestra.run.smithi047.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-01-18T06:46:03.218 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-01-18T06:46:03.218 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-01-18T06:46:03.219 INFO:teuthology.orchestra.run.smithi047.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-01-18T06:46:03.219 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-18T06:46:03.219 INFO:teuthology.orchestra.run.smithi047.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-18T06:46:03.219 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:46:03.219 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-01-18T06:46:03.219 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:46:03.219 INFO:teuthology.orchestra.run.smithi047.stdout:Install 13 Packages 2024-01-18T06:46:03.219 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:46:03.220 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 1.9 M 2024-01-18T06:46:03.220 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 24 M 2024-01-18T06:46:03.220 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-01-18T06:46:03.318 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-01-18T06:46:03.399 INFO:teuthology.orchestra.run.smithi047.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 633 kB/s | 27 kB 00:00 2024-01-18T06:46:03.425 INFO:teuthology.orchestra.run.smithi047.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 392 kB/s | 27 kB 00:00 2024-01-18T06:46:03.450 INFO:teuthology.orchestra.run.smithi047.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 215 kB/s | 20 kB 00:00 2024-01-18T06:46:03.525 INFO:teuthology.orchestra.run.smithi047.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 1.4 MB/s | 110 kB 00:00 2024-01-18T06:46:03.550 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-01-18T06:46:03.550 INFO:teuthology.orchestra.run.smithi047.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 837 kB/s | 126 kB 00:00 2024-01-18T06:46:03.576 INFO:teuthology.orchestra.run.smithi047.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 1.2 MB/s | 155 kB 00:00 2024-01-18T06:46:03.601 INFO:teuthology.orchestra.run.smithi047.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 563 kB/s | 43 kB 00:00 2024-01-18T06:46:03.752 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-01-18T06:46:03.902 INFO:teuthology.orchestra.run.smithi047.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 50 kB/s | 16 kB 00:00 2024-01-18T06:46:03.960 INFO:teuthology.orchestra.run.smithi047.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 81 kB/s | 33 kB 00:00 2024-01-18T06:46:03.987 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-01-18T06:46:04.095 INFO:teuthology.orchestra.run.smithi047.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 165 kB/s | 81 kB 00:00 2024-01-18T06:46:04.169 INFO:teuthology.orchestra.run.smithi047.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 308 kB/s | 64 kB 00:00 2024-01-18T06:46:04.234 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-01-18T06:46:04.294 INFO:teuthology.orchestra.run.smithi047.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 306 kB/s | 61 kB 00:00 2024-01-18T06:46:04.434 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-01-18T06:46:04.705 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-01-18T06:46:04.987 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-01-18T06:46:05.247 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-01-18T06:46:05.928 INFO:teuthology.orchestra.run.smithi047.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 589 kB/s | 1.2 MB 00:02 2024-01-18T06:46:05.929 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-01-18T06:46:05.929 INFO:teuthology.orchestra.run.smithi047.stdout:Total 722 kB/s | 1.9 MB 00:02 2024-01-18T06:46:05.987 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-01-18T06:46:06.004 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-01-18T06:46:06.005 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-01-18T06:46:06.100 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-01-18T06:46:06.104 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-01-18T06:46:06.653 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-01-18T06:46:06.743 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-01-18T06:46:06.913 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-01-18T06:46:07.008 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-01-18T06:46:07.089 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-01-18T06:46:07.217 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-01-18T06:46:07.336 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-01-18T06:46:07.443 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-01-18T06:46:07.598 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-01-18T06:46:07.618 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-18T06:46:07.823 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-01-18T06:46:07.906 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-18T06:46:07.907 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-01-18T06:46:07.907 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-01-18T06:46:07.907 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-01-18T06:46:07.907 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-01-18T06:46:07.907 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-01-18T06:46:07.907 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-01-18T06:46:07.907 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-01-18T06:46:07.907 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-01-18T06:46:07.907 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-01-18T06:46:07.907 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-01-18T06:46:07.907 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-01-18T06:46:07.907 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-01-18T06:46:08.076 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-01-18T06:46:08.388 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-01-18T06:46:08.388 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:46:08.388 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-01-18T06:46:08.388 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-01-18T06:46:08.388 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-18T06:46:08.389 INFO:teuthology.orchestra.run.smithi194.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-18T06:46:08.389 INFO:teuthology.orchestra.run.smithi194.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-18T06:46:08.389 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-18T06:46:08.389 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-18T06:46:08.389 INFO:teuthology.orchestra.run.smithi194.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-18T06:46:08.389 INFO:teuthology.orchestra.run.smithi194.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-18T06:46:08.389 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-18T06:46:08.389 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-18T06:46:08.389 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-18T06:46:08.389 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-18T06:46:08.389 INFO:teuthology.orchestra.run.smithi194.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-18T06:46:08.389 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:46:08.389 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-18T06:46:08.414 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-01-18T06:46:08.498 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr-cephadm 2024-01-18T06:46:08.682 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-01-18T06:46:09.227 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:51 ago on Thu 18 Jan 2024 06:44:18 AM UTC. 2024-01-18T06:46:09.840 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-01-18T06:46:10.131 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-01-18T06:46:10.374 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-01-18T06:46:10.574 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-01-18T06:46:10.709 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-18T06:46:10.710 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:46:10.710 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-01-18T06:46:10.710 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:46:10.710 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-01-18T06:46:10.710 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-01-18T06:46:10.710 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-01-18T06:46:10.710 INFO:teuthology.orchestra.run.smithi194.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-01-18T06:46:10.710 INFO:teuthology.orchestra.run.smithi194.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-01-18T06:46:10.710 INFO:teuthology.orchestra.run.smithi194.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-01-18T06:46:10.710 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-01-18T06:46:10.711 INFO:teuthology.orchestra.run.smithi194.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-01-18T06:46:10.711 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:46:10.711 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-01-18T06:46:10.711 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:46:10.711 INFO:teuthology.orchestra.run.smithi194.stdout:Install 6 Packages 2024-01-18T06:46:10.711 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:46:10.711 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 5.8 M 2024-01-18T06:46:10.711 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 24 M 2024-01-18T06:46:10.712 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-01-18T06:46:10.759 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-18T06:46:10.981 INFO:teuthology.orchestra.run.smithi194.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 2.2 MB/s | 89 kB 00:00 2024-01-18T06:46:11.032 INFO:teuthology.orchestra.run.smithi194.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 5.8 MB/s | 538 kB 00:00 2024-01-18T06:46:11.069 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-18T06:46:11.069 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-01-18T06:46:11.070 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-01-18T06:46:11.070 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-01-18T06:46:11.070 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-01-18T06:46:11.070 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-01-18T06:46:11.070 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-01-18T06:46:11.070 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-01-18T06:46:11.070 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-01-18T06:46:11.070 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-01-18T06:46:11.070 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-01-18T06:46:11.070 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-01-18T06:46:11.070 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-01-18T06:46:11.132 INFO:teuthology.orchestra.run.smithi194.stdout:(3/6): python3-babel-2.5.1-7.el8.noarch.rpm 25 MB/s | 4.8 MB 00:00 2024-01-18T06:46:11.341 INFO:teuthology.orchestra.run.smithi194.stdout:(4/6): python3-apipkg-1.5-6.el8.noarch.rpm 59 kB/s | 21 kB 00:00 2024-01-18T06:46:11.541 INFO:teuthology.orchestra.run.smithi194.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 120 kB/s | 49 kB 00:00 2024-01-18T06:46:11.554 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-01-18T06:46:11.554 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:46:11.555 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-01-18T06:46:11.555 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-01-18T06:46:11.555 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-18T06:46:11.555 INFO:teuthology.orchestra.run.smithi047.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-18T06:46:11.555 INFO:teuthology.orchestra.run.smithi047.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-18T06:46:11.555 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-18T06:46:11.555 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-18T06:46:11.555 INFO:teuthology.orchestra.run.smithi047.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-18T06:46:11.555 INFO:teuthology.orchestra.run.smithi047.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-18T06:46:11.555 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-18T06:46:11.555 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-18T06:46:11.555 INFO:teuthology.orchestra.run.smithi047.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-18T06:46:11.555 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-18T06:46:11.556 INFO:teuthology.orchestra.run.smithi047.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-18T06:46:11.556 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:46:11.556 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-01-18T06:46:11.698 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-mgr-cephadm 2024-01-18T06:46:12.066 INFO:teuthology.orchestra.run.smithi194.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 344 kB/s | 355 kB 00:01 2024-01-18T06:46:12.067 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-01-18T06:46:12.067 INFO:teuthology.orchestra.run.smithi194.stdout:Total 4.3 MB/s | 5.8 MB 00:01 2024-01-18T06:46:12.120 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-01-18T06:46:12.130 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-01-18T06:46:12.131 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-01-18T06:46:12.237 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-01-18T06:46:12.240 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-01-18T06:46:12.455 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:58 ago on Thu 18 Jan 2024 06:44:14 AM UTC. 2024-01-18T06:46:12.515 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-01-18T06:46:12.792 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-01-18T06:46:13.086 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-01-18T06:46:13.989 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-01-18T06:46:14.039 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-01-18T06:46:14.040 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:46:14.040 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-01-18T06:46:14.040 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:46:14.040 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-01-18T06:46:14.040 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-01-18T06:46:14.040 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-01-18T06:46:14.040 INFO:teuthology.orchestra.run.smithi047.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-01-18T06:46:14.040 INFO:teuthology.orchestra.run.smithi047.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-01-18T06:46:14.040 INFO:teuthology.orchestra.run.smithi047.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-01-18T06:46:14.040 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-01-18T06:46:14.040 INFO:teuthology.orchestra.run.smithi047.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-01-18T06:46:14.040 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:46:14.041 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-01-18T06:46:14.041 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:46:14.041 INFO:teuthology.orchestra.run.smithi047.stdout:Install 6 Packages 2024-01-18T06:46:14.041 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:46:14.041 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 5.8 M 2024-01-18T06:46:14.041 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 24 M 2024-01-18T06:46:14.041 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-01-18T06:46:14.329 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-01-18T06:46:14.547 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-01-18T06:46:14.717 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-18T06:46:15.008 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-18T06:46:15.008 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-01-18T06:46:15.008 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-01-18T06:46:15.008 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-01-18T06:46:15.008 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-01-18T06:46:15.008 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-01-18T06:46:15.417 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-01-18T06:46:15.417 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:46:15.417 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-01-18T06:46:15.417 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-01-18T06:46:15.417 INFO:teuthology.orchestra.run.smithi194.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-01-18T06:46:15.417 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-01-18T06:46:15.417 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:46:15.417 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-18T06:46:15.542 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-fuse 2024-01-18T06:46:16.267 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:58 ago on Thu 18 Jan 2024 06:44:18 AM UTC. 2024-01-18T06:46:17.751 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-18T06:46:17.752 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:46:17.752 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-01-18T06:46:17.752 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:46:17.752 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-01-18T06:46:17.752 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-01-18T06:46:17.752 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-01-18T06:46:17.752 INFO:teuthology.orchestra.run.smithi194.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-01-18T06:46:17.752 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:46:17.752 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-01-18T06:46:17.752 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:46:17.752 INFO:teuthology.orchestra.run.smithi194.stdout:Install 2 Packages 2024-01-18T06:46:17.753 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:46:17.753 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 968 k 2024-01-18T06:46:17.753 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 3.2 M 2024-01-18T06:46:17.753 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-01-18T06:46:17.876 INFO:teuthology.orchestra.run.smithi194.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 11 MB/s | 885 kB 00:00 2024-01-18T06:46:17.910 INFO:teuthology.orchestra.run.smithi194.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 717 kB/s | 83 kB 00:00 2024-01-18T06:46:17.910 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-01-18T06:46:17.911 INFO:teuthology.orchestra.run.smithi194.stdout:Total 6.0 MB/s | 968 kB 00:00 2024-01-18T06:46:17.920 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-01-18T06:46:17.940 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-01-18T06:46:17.941 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-01-18T06:46:18.026 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-01-18T06:46:18.029 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-01-18T06:46:18.207 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-01-18T06:46:18.475 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-01-18T06:46:18.692 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-18T06:46:18.802 INFO:teuthology.orchestra.run.smithi047.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 2.0 MB/s | 89 kB 00:00 2024-01-18T06:46:18.852 INFO:teuthology.orchestra.run.smithi047.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 5.7 MB/s | 538 kB 00:00 2024-01-18T06:46:18.960 INFO:teuthology.orchestra.run.smithi047.stdout:(3/6): python3-babel-2.5.1-7.el8.noarch.rpm 24 MB/s | 4.8 MB 00:00 2024-01-18T06:46:19.091 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-18T06:46:19.091 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-01-18T06:46:19.267 INFO:teuthology.orchestra.run.smithi047.stdout:(4/6): python3-apipkg-1.5-6.el8.noarch.rpm 46 kB/s | 21 kB 00:00 2024-01-18T06:46:19.494 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-01-18T06:46:19.494 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:46:19.494 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-01-18T06:46:19.494 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-01-18T06:46:19.494 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:46:19.494 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-18T06:46:19.509 INFO:teuthology.orchestra.run.smithi047.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 89 kB/s | 49 kB 00:00 2024-01-18T06:46:19.705 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install librados-devel 2024-01-18T06:46:19.763 INFO:teuthology.orchestra.run.smithi047.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 390 kB/s | 355 kB 00:00 2024-01-18T06:46:19.765 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-01-18T06:46:19.765 INFO:teuthology.orchestra.run.smithi047.stdout:Total 1.0 MB/s | 5.8 MB 00:05 2024-01-18T06:46:19.820 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-01-18T06:46:19.831 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-01-18T06:46:19.831 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-01-18T06:46:19.959 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-01-18T06:46:19.963 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-01-18T06:46:20.242 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-01-18T06:46:20.444 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:02:02 ago on Thu 18 Jan 2024 06:44:18 AM UTC. 2024-01-18T06:46:20.486 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-01-18T06:46:20.811 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-01-18T06:46:21.810 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-01-18T06:46:21.920 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-18T06:46:21.921 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:46:21.921 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-01-18T06:46:21.921 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:46:21.921 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-01-18T06:46:21.921 INFO:teuthology.orchestra.run.smithi194.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-01-18T06:46:21.921 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:46:21.921 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-01-18T06:46:21.921 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:46:21.921 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-01-18T06:46:21.921 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:46:21.922 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 131 k 2024-01-18T06:46:21.922 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 439 k 2024-01-18T06:46:21.922 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-01-18T06:46:21.963 INFO:teuthology.orchestra.run.smithi194.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 3.1 MB/s | 131 kB 00:00 2024-01-18T06:46:21.964 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-01-18T06:46:21.964 INFO:teuthology.orchestra.run.smithi194.stdout:Total 3.0 MB/s | 131 kB 00:00 2024-01-18T06:46:21.964 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-01-18T06:46:21.969 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-01-18T06:46:21.969 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-01-18T06:46:22.020 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-01-18T06:46:22.023 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-01-18T06:46:22.144 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-01-18T06:46:22.149 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-01-18T06:46:22.345 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-01-18T06:46:22.365 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T06:46:22.564 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-18T06:46:22.648 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T06:46:22.914 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-18T06:46:22.914 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-01-18T06:46:22.914 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-01-18T06:46:22.915 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-01-18T06:46:22.915 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-01-18T06:46:22.915 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-01-18T06:46:23.033 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T06:46:23.034 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:46:23.034 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-01-18T06:46:23.034 INFO:teuthology.orchestra.run.smithi194.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-01-18T06:46:23.034 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:46:23.034 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-18T06:46:23.238 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install libcephfs2 2024-01-18T06:46:23.344 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-01-18T06:46:23.344 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:46:23.344 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-01-18T06:46:23.344 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-01-18T06:46:23.344 INFO:teuthology.orchestra.run.smithi047.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-01-18T06:46:23.344 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-01-18T06:46:23.345 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:46:23.345 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-01-18T06:46:23.474 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-fuse 2024-01-18T06:46:23.973 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:02:05 ago on Thu 18 Jan 2024 06:44:18 AM UTC. 2024-01-18T06:46:24.231 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:02:10 ago on Thu 18 Jan 2024 06:44:14 AM UTC. 2024-01-18T06:46:25.408 INFO:teuthology.orchestra.run.smithi194.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-18T06:46:25.453 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-18T06:46:25.455 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-01-18T06:46:25.455 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-18T06:46:25.542 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install libcephfs-devel 2024-01-18T06:46:25.832 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-01-18T06:46:25.833 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:46:25.833 INFO:teuthology.orchestra.run.smithi047.stdout: Package Architecture Version Repository Size 2024-01-18T06:46:25.833 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:46:25.833 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-01-18T06:46:25.833 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-01-18T06:46:25.833 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-01-18T06:46:25.833 INFO:teuthology.orchestra.run.smithi047.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-01-18T06:46:25.833 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:46:25.833 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-01-18T06:46:25.833 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:46:25.833 INFO:teuthology.orchestra.run.smithi047.stdout:Install 2 Packages 2024-01-18T06:46:25.833 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:46:25.834 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 968 k 2024-01-18T06:46:25.834 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 3.2 M 2024-01-18T06:46:25.834 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-01-18T06:46:26.082 INFO:teuthology.orchestra.run.smithi047.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 15 MB/s | 885 kB 00:00 2024-01-18T06:46:26.107 INFO:teuthology.orchestra.run.smithi047.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 1.0 MB/s | 83 kB 00:00 2024-01-18T06:46:26.108 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-01-18T06:46:26.108 INFO:teuthology.orchestra.run.smithi047.stdout:Total 3.4 MB/s | 968 kB 00:00 2024-01-18T06:46:26.118 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-01-18T06:46:26.140 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-01-18T06:46:26.140 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-01-18T06:46:26.234 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-01-18T06:46:26.238 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-01-18T06:46:26.267 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:02:08 ago on Thu 18 Jan 2024 06:44:18 AM UTC. 2024-01-18T06:46:26.428 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-01-18T06:46:26.751 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-01-18T06:46:26.981 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-18T06:46:27.392 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-18T06:46:27.392 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-01-18T06:46:27.765 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-18T06:46:27.765 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:46:27.766 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-01-18T06:46:27.766 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:46:27.766 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-01-18T06:46:27.766 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-01-18T06:46:27.766 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:46:27.766 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-01-18T06:46:27.766 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:46:27.766 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-01-18T06:46:27.766 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:46:27.767 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 24 k 2024-01-18T06:46:27.767 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 94 k 2024-01-18T06:46:27.767 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-01-18T06:46:27.810 INFO:teuthology.orchestra.run.smithi194.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 558 kB/s | 24 kB 00:00 2024-01-18T06:46:27.811 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-01-18T06:46:27.811 INFO:teuthology.orchestra.run.smithi194.stdout:Total 542 kB/s | 24 kB 00:00 2024-01-18T06:46:27.811 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-01-18T06:46:27.813 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-01-18T06:46:27.813 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:46:27.813 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-01-18T06:46:27.813 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-01-18T06:46:27.813 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:46:27.814 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-01-18T06:46:27.815 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-01-18T06:46:27.815 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-01-18T06:46:27.817 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-01-18T06:46:27.820 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-01-18T06:46:27.872 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-01-18T06:46:28.047 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T06:46:28.057 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install librados-devel 2024-01-18T06:46:28.319 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T06:46:28.665 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T06:46:28.666 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:46:28.666 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-01-18T06:46:28.666 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-01-18T06:46:28.666 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:46:28.666 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-18T06:46:28.775 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install librados2 2024-01-18T06:46:28.817 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:02:14 ago on Thu 18 Jan 2024 06:44:14 AM UTC. 2024-01-18T06:46:29.499 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:02:11 ago on Thu 18 Jan 2024 06:44:18 AM UTC. 2024-01-18T06:46:30.401 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-01-18T06:46:30.402 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:46:30.402 INFO:teuthology.orchestra.run.smithi047.stdout: Package Architecture Version Repository Size 2024-01-18T06:46:30.402 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:46:30.402 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-01-18T06:46:30.402 INFO:teuthology.orchestra.run.smithi047.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-01-18T06:46:30.402 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:46:30.402 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-01-18T06:46:30.402 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:46:30.402 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-01-18T06:46:30.402 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:46:30.403 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 131 k 2024-01-18T06:46:30.403 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 439 k 2024-01-18T06:46:30.403 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-01-18T06:46:30.671 INFO:teuthology.orchestra.run.smithi047.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 490 kB/s | 131 kB 00:00 2024-01-18T06:46:30.672 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-01-18T06:46:30.672 INFO:teuthology.orchestra.run.smithi047.stdout:Total 488 kB/s | 131 kB 00:00 2024-01-18T06:46:30.672 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-01-18T06:46:30.678 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-01-18T06:46:30.678 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-01-18T06:46:30.732 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-01-18T06:46:30.735 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-01-18T06:46:30.933 INFO:teuthology.orchestra.run.smithi194.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-18T06:46:30.979 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-18T06:46:30.980 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-01-18T06:46:30.980 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-18T06:46:31.075 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install librbd1 2024-01-18T06:46:31.213 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-01-18T06:46:31.779 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T06:46:31.812 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:02:13 ago on Thu 18 Jan 2024 06:44:18 AM UTC. 2024-01-18T06:46:32.096 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T06:46:32.467 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T06:46:32.467 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:46:32.468 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-01-18T06:46:32.468 INFO:teuthology.orchestra.run.smithi047.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-01-18T06:46:32.468 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:46:32.468 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-01-18T06:46:32.678 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install libcephfs2 2024-01-18T06:46:33.234 INFO:teuthology.orchestra.run.smithi194.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-18T06:46:33.280 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-18T06:46:33.281 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-01-18T06:46:33.281 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-18T06:46:33.374 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-rados 2024-01-18T06:46:33.438 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:02:19 ago on Thu 18 Jan 2024 06:44:14 AM UTC. 2024-01-18T06:46:34.099 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:02:16 ago on Thu 18 Jan 2024 06:44:18 AM UTC. 2024-01-18T06:46:34.994 INFO:teuthology.orchestra.run.smithi047.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-18T06:46:35.043 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-01-18T06:46:35.044 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-01-18T06:46:35.044 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-01-18T06:46:35.141 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install libcephfs-devel 2024-01-18T06:46:35.516 INFO:teuthology.orchestra.run.smithi194.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-18T06:46:35.561 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-18T06:46:35.563 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-01-18T06:46:35.563 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-18T06:46:35.652 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-rgw 2024-01-18T06:46:35.901 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:02:21 ago on Thu 18 Jan 2024 06:44:14 AM UTC. 2024-01-18T06:46:36.380 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:02:18 ago on Thu 18 Jan 2024 06:44:18 AM UTC. 2024-01-18T06:46:37.489 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-01-18T06:46:37.489 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:46:37.489 INFO:teuthology.orchestra.run.smithi047.stdout: Package Architecture Version Repository Size 2024-01-18T06:46:37.489 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:46:37.489 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-01-18T06:46:37.489 INFO:teuthology.orchestra.run.smithi047.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-01-18T06:46:37.490 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:46:37.490 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-01-18T06:46:37.490 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:46:37.490 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-01-18T06:46:37.490 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:46:37.490 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 24 k 2024-01-18T06:46:37.490 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 94 k 2024-01-18T06:46:37.490 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-01-18T06:46:37.542 INFO:teuthology.orchestra.run.smithi047.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 470 kB/s | 24 kB 00:00 2024-01-18T06:46:37.543 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-01-18T06:46:37.543 INFO:teuthology.orchestra.run.smithi047.stdout:Total 459 kB/s | 24 kB 00:00 2024-01-18T06:46:37.543 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-01-18T06:46:37.547 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-01-18T06:46:37.547 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-01-18T06:46:37.549 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-01-18T06:46:37.553 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-01-18T06:46:37.600 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-01-18T06:46:37.828 INFO:teuthology.orchestra.run.smithi194.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-18T06:46:37.873 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-18T06:46:37.874 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-01-18T06:46:37.874 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-18T06:46:37.953 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T06:46:37.964 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-cephfs 2024-01-18T06:46:38.240 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T06:46:38.692 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:02:20 ago on Thu 18 Jan 2024 06:44:18 AM UTC. 2024-01-18T06:46:38.741 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T06:46:38.741 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:46:38.741 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-01-18T06:46:38.741 INFO:teuthology.orchestra.run.smithi047.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-01-18T06:46:38.742 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:46:38.742 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-01-18T06:46:38.997 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install librados2 2024-01-18T06:46:39.756 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:02:25 ago on Thu 18 Jan 2024 06:44:14 AM UTC. 2024-01-18T06:46:40.103 INFO:teuthology.orchestra.run.smithi194.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-18T06:46:40.148 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-18T06:46:40.149 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-01-18T06:46:40.149 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-18T06:46:40.243 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-rbd 2024-01-18T06:46:40.972 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:02:22 ago on Thu 18 Jan 2024 06:44:18 AM UTC. 2024-01-18T06:46:41.308 INFO:teuthology.orchestra.run.smithi047.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-18T06:46:41.358 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-01-18T06:46:41.359 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-01-18T06:46:41.360 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-01-18T06:46:41.448 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install librbd1 2024-01-18T06:46:42.208 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:02:28 ago on Thu 18 Jan 2024 06:44:14 AM UTC. 2024-01-18T06:46:42.420 INFO:teuthology.orchestra.run.smithi194.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-18T06:46:42.465 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-18T06:46:42.466 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-01-18T06:46:42.466 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-18T06:46:42.559 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install rbd-fuse 2024-01-18T06:46:43.284 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:02:25 ago on Thu 18 Jan 2024 06:44:18 AM UTC. 2024-01-18T06:46:43.764 INFO:teuthology.orchestra.run.smithi047.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-18T06:46:43.813 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-01-18T06:46:43.814 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-01-18T06:46:43.814 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-01-18T06:46:43.924 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install python3-rados 2024-01-18T06:46:44.682 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:02:30 ago on Thu 18 Jan 2024 06:44:14 AM UTC. 2024-01-18T06:46:44.776 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-18T06:46:44.777 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:46:44.777 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-01-18T06:46:44.777 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:46:44.777 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-01-18T06:46:44.777 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-01-18T06:46:44.777 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:46:44.777 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-01-18T06:46:44.777 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:46:44.777 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-01-18T06:46:44.777 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:46:44.778 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 89 k 2024-01-18T06:46:44.778 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 298 k 2024-01-18T06:46:44.778 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-01-18T06:46:44.816 INFO:teuthology.orchestra.run.smithi194.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 2.3 MB/s | 89 kB 00:00 2024-01-18T06:46:44.816 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-01-18T06:46:44.816 INFO:teuthology.orchestra.run.smithi194.stdout:Total 2.2 MB/s | 89 kB 00:00 2024-01-18T06:46:44.817 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-01-18T06:46:44.826 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-01-18T06:46:44.826 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-01-18T06:46:44.875 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-01-18T06:46:44.877 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-01-18T06:46:44.980 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-01-18T06:46:45.211 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T06:46:45.507 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T06:46:45.852 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T06:46:45.852 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:46:45.852 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-01-18T06:46:45.852 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-01-18T06:46:45.852 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:46:45.852 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-18T06:46:46.052 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install rbd-mirror 2024-01-18T06:46:46.198 INFO:teuthology.orchestra.run.smithi047.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-18T06:46:46.247 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-01-18T06:46:46.248 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-01-18T06:46:46.249 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-01-18T06:46:46.348 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install python3-rgw 2024-01-18T06:46:46.792 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:02:28 ago on Thu 18 Jan 2024 06:44:18 AM UTC. 2024-01-18T06:46:47.117 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:02:33 ago on Thu 18 Jan 2024 06:44:14 AM UTC. 2024-01-18T06:46:48.282 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-18T06:46:48.283 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:46:48.283 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-01-18T06:46:48.283 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:46:48.283 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-01-18T06:46:48.283 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-18T06:46:48.283 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:46:48.283 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-01-18T06:46:48.284 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:46:48.284 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-01-18T06:46:48.284 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:46:48.284 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 3.7 M 2024-01-18T06:46:48.284 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 18 M 2024-01-18T06:46:48.284 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-01-18T06:46:48.376 INFO:teuthology.orchestra.run.smithi194.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 40 MB/s | 3.7 MB 00:00 2024-01-18T06:46:48.377 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-01-18T06:46:48.377 INFO:teuthology.orchestra.run.smithi194.stdout:Total 40 MB/s | 3.7 MB 00:00 2024-01-18T06:46:48.378 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-01-18T06:46:48.389 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-01-18T06:46:48.389 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-01-18T06:46:48.452 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-01-18T06:46:48.455 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-01-18T06:46:48.649 INFO:teuthology.orchestra.run.smithi047.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-18T06:46:48.698 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-01-18T06:46:48.699 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-01-18T06:46:48.699 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-01-18T06:46:48.792 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install python3-cephfs 2024-01-18T06:46:48.922 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-01-18T06:46:49.170 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T06:46:49.548 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:02:35 ago on Thu 18 Jan 2024 06:44:14 AM UTC. 2024-01-18T06:46:49.582 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T06:46:49.982 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T06:46:49.982 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:46:49.982 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-01-18T06:46:49.982 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-01-18T06:46:49.982 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:46:49.983 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-18T06:46:50.188 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install rbd-nbd 2024-01-18T06:46:50.924 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:02:32 ago on Thu 18 Jan 2024 06:44:18 AM UTC. 2024-01-18T06:46:51.094 INFO:teuthology.orchestra.run.smithi047.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-18T06:46:51.144 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-01-18T06:46:51.145 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-01-18T06:46:51.145 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-01-18T06:46:51.241 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install python3-rbd 2024-01-18T06:46:51.995 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:02:37 ago on Thu 18 Jan 2024 06:44:14 AM UTC. 2024-01-18T06:46:52.418 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-18T06:46:52.418 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:46:52.419 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-01-18T06:46:52.419 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:46:52.419 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-01-18T06:46:52.419 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-01-18T06:46:52.419 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:46:52.419 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-01-18T06:46:52.419 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-18T06:46:52.419 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-01-18T06:46:52.419 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:46:52.420 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 173 k 2024-01-18T06:46:52.420 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 649 k 2024-01-18T06:46:52.420 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-01-18T06:46:52.463 INFO:teuthology.orchestra.run.smithi194.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 4.0 MB/s | 173 kB 00:00 2024-01-18T06:46:52.463 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-01-18T06:46:52.463 INFO:teuthology.orchestra.run.smithi194.stdout:Total 3.9 MB/s | 173 kB 00:00 2024-01-18T06:46:52.463 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-01-18T06:46:52.474 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-01-18T06:46:52.474 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-01-18T06:46:52.524 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-01-18T06:46:52.527 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-01-18T06:46:52.639 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-01-18T06:46:52.843 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T06:46:53.141 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T06:46:53.520 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T06:46:53.520 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:46:53.520 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-01-18T06:46:53.521 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-01-18T06:46:53.521 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:46:53.521 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-18T06:46:53.533 INFO:teuthology.orchestra.run.smithi047.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-18T06:46:53.582 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-01-18T06:46:53.584 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-01-18T06:46:53.584 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-01-18T06:46:53.669 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install rbd-fuse 2024-01-18T06:46:53.742 DEBUG:teuthology.parallel:result is None 2024-01-18T06:46:54.422 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:02:40 ago on Thu 18 Jan 2024 06:44:14 AM UTC. 2024-01-18T06:46:56.009 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-01-18T06:46:56.009 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:46:56.010 INFO:teuthology.orchestra.run.smithi047.stdout: Package Architecture Version Repository Size 2024-01-18T06:46:56.010 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:46:56.010 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-01-18T06:46:56.010 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-01-18T06:46:56.010 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:46:56.010 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-01-18T06:46:56.010 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:46:56.010 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-01-18T06:46:56.010 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:46:56.010 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 89 k 2024-01-18T06:46:56.010 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 298 k 2024-01-18T06:46:56.010 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-01-18T06:46:56.057 INFO:teuthology.orchestra.run.smithi047.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.8 MB/s | 89 kB 00:00 2024-01-18T06:46:56.058 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-01-18T06:46:56.058 INFO:teuthology.orchestra.run.smithi047.stdout:Total 1.8 MB/s | 89 kB 00:00 2024-01-18T06:46:56.058 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-01-18T06:46:56.069 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-01-18T06:46:56.069 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-01-18T06:46:56.122 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-01-18T06:46:56.125 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-01-18T06:46:56.229 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-01-18T06:46:56.434 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T06:46:56.756 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T06:46:57.197 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T06:46:57.197 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:46:57.197 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-01-18T06:46:57.197 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-01-18T06:46:57.197 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:46:57.197 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-01-18T06:46:57.419 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install rbd-mirror 2024-01-18T06:46:58.178 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:02:44 ago on Thu 18 Jan 2024 06:44:14 AM UTC. 2024-01-18T06:46:59.770 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-01-18T06:46:59.771 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:46:59.771 INFO:teuthology.orchestra.run.smithi047.stdout: Package Architecture Version Repository Size 2024-01-18T06:46:59.771 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:46:59.771 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-01-18T06:46:59.771 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-18T06:46:59.771 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:46:59.771 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-01-18T06:46:59.771 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:46:59.771 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-01-18T06:46:59.771 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:46:59.772 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 3.7 M 2024-01-18T06:46:59.772 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 18 M 2024-01-18T06:46:59.772 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-01-18T06:46:59.853 INFO:teuthology.orchestra.run.smithi047.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 46 MB/s | 3.7 MB 00:00 2024-01-18T06:46:59.853 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-01-18T06:46:59.854 INFO:teuthology.orchestra.run.smithi047.stdout:Total 45 MB/s | 3.7 MB 00:00 2024-01-18T06:46:59.854 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-01-18T06:46:59.867 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-01-18T06:46:59.867 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-01-18T06:46:59.934 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-01-18T06:46:59.937 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-01-18T06:47:00.428 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-01-18T06:47:00.668 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T06:47:01.101 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T06:47:01.495 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T06:47:01.496 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:47:01.496 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-01-18T06:47:01.496 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-01-18T06:47:01.496 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:47:01.496 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-01-18T06:47:01.734 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install rbd-nbd 2024-01-18T06:47:02.502 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:02:48 ago on Thu 18 Jan 2024 06:44:14 AM UTC. 2024-01-18T06:47:04.106 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-01-18T06:47:04.107 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:47:04.107 INFO:teuthology.orchestra.run.smithi047.stdout: Package Architecture Version Repository Size 2024-01-18T06:47:04.107 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:47:04.107 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-01-18T06:47:04.107 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-01-18T06:47:04.107 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:47:04.107 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-01-18T06:47:04.107 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-01-18T06:47:04.107 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-01-18T06:47:04.107 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:47:04.108 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 173 k 2024-01-18T06:47:04.108 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 649 k 2024-01-18T06:47:04.108 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-01-18T06:47:04.156 INFO:teuthology.orchestra.run.smithi047.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.6 MB/s | 173 kB 00:00 2024-01-18T06:47:04.157 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-01-18T06:47:04.157 INFO:teuthology.orchestra.run.smithi047.stdout:Total 3.5 MB/s | 173 kB 00:00 2024-01-18T06:47:04.157 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-01-18T06:47:04.168 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-01-18T06:47:04.169 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-01-18T06:47:04.224 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-01-18T06:47:04.227 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-01-18T06:47:04.348 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-01-18T06:47:04.566 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T06:47:04.867 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T06:47:05.285 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T06:47:05.286 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:47:05.286 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-01-18T06:47:05.286 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-01-18T06:47:05.286 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-01-18T06:47:05.286 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-01-18T06:47:05.508 DEBUG:teuthology.parallel:result is None 2024-01-18T06:47:05.508 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-18T06:47:05.508 INFO:teuthology.packaging:ref: None 2024-01-18T06:47:05.509 INFO:teuthology.packaging:tag: v16.2.4 2024-01-18T06:47:05.509 INFO:teuthology.packaging:branch: None 2024-01-18T06:47:05.509 INFO:teuthology.packaging:sha1: 491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0 2024-01-18T06:47:05.509 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-18T06:47:05.724 DEBUG:teuthology.orchestra.run.smithi047:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-18T06:47:05.747 INFO:teuthology.orchestra.run.smithi047.stdout:16.2.4-0.el8 2024-01-18T06:47:05.748 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-01-18T06:47:05.748 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-01-18T06:47:05.750 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-18T06:47:05.750 INFO:teuthology.packaging:ref: None 2024-01-18T06:47:05.750 INFO:teuthology.packaging:tag: v16.2.4 2024-01-18T06:47:05.750 INFO:teuthology.packaging:branch: None 2024-01-18T06:47:05.750 INFO:teuthology.packaging:sha1: 491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0 2024-01-18T06:47:05.750 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-18T06:47:05.887 DEBUG:teuthology.orchestra.run.smithi194:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-18T06:47:05.914 INFO:teuthology.orchestra.run.smithi194.stdout:16.2.4-0.el8 2024-01-18T06:47:05.914 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-01-18T06:47:05.914 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-01-18T06:47:05.916 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-18T06:47:05.916 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-01-18T06:47:05.916 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-18T06:47:05.947 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-01-18T06:47:05.947 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-18T06:47:05.986 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-18T06:47:05.987 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-01-18T06:47:05.987 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/usr/bin/daemon-helper 2024-01-18T06:47:06.016 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-18T06:47:06.084 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-01-18T06:47:06.085 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/daemon-helper 2024-01-18T06:47:06.114 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-18T06:47:06.183 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-18T06:47:06.183 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-01-18T06:47:06.183 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-18T06:47:06.212 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-18T06:47:06.280 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-01-18T06:47:06.281 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-18T06:47:06.309 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-18T06:47:06.378 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-18T06:47:06.378 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-01-18T06:47:06.378 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/usr/bin/stdin-killer 2024-01-18T06:47:06.407 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-18T06:47:06.475 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-01-18T06:47:06.475 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/stdin-killer 2024-01-18T06:47:06.502 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-18T06:47:06.570 INFO:teuthology.run_tasks:Running task print... 2024-01-18T06:47:06.580 INFO:teuthology.task.print:**** done install task... 2024-01-18T06:47:06.580 INFO:teuthology.run_tasks:Running task cephadm... 2024-01-18T06:47:06.655 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0'} 2024-01-18T06:47:06.656 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2024-01-18T06:47:06.656 INFO:tasks.cephadm:Cluster fsid is 653e4012-b5cd-11ee-95ae-87774f69a715 2024-01-18T06:47:06.656 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-01-18T06:47:06.656 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-01-18T06:47:06.657 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi047': '172.21.15.47', 'mon.smithi194': '172.21.15.194'} 2024-01-18T06:47:06.657 INFO:tasks.cephadm:Normalizing hostnames... 2024-01-18T06:47:06.657 DEBUG:teuthology.orchestra.run.smithi047:> sudo hostname $(hostname -s) 2024-01-18T06:47:06.686 DEBUG:teuthology.orchestra.run.smithi194:> sudo hostname $(hostname -s) 2024-01-18T06:47:06.714 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2024-01-18T06:47:06.714 DEBUG:teuthology.orchestra.run.smithi047:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-18T06:47:06.960 INFO:teuthology.orchestra.run.smithi047.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Jan 18 06:47 /home/ubuntu/cephtest/cephadm 2024-01-18T06:47:06.960 DEBUG:teuthology.orchestra.run.smithi194:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-18T06:47:07.194 INFO:teuthology.orchestra.run.smithi194.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Jan 18 06:47 /home/ubuntu/cephtest/cephadm 2024-01-18T06:47:07.194 DEBUG:teuthology.orchestra.run.smithi047:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-18T06:47:07.219 DEBUG:teuthology.orchestra.run.smithi194:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-18T06:47:07.251 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /etc/ceph 2024-01-18T06:47:07.293 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /etc/ceph 2024-01-18T06:47:07.323 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 777 /etc/ceph 2024-01-18T06:47:07.364 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 777 /etc/ceph 2024-01-18T06:47:07.392 INFO:tasks.cephadm:Writing seed config... 2024-01-18T06:47:07.393 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-01-18T06:47:07.393 INFO:tasks.cephadm: override: [client] debug client = 20 2024-01-18T06:47:07.393 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-01-18T06:47:07.393 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-01-18T06:47:07.393 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-01-18T06:47:07.393 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-01-18T06:47:07.393 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-01-18T06:47:07.393 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-01-18T06:47:07.393 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-01-18T06:47:07.393 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-01-18T06:47:07.393 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-01-18T06:47:07.393 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-01-18T06:47:07.393 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-01-18T06:47:07.393 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-01-18T06:47:07.394 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-01-18T06:47:07.394 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-01-18T06:47:07.394 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-01-18T06:47:07.394 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-01-18T06:47:07.394 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-01-18T06:47:07.394 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-01-18T06:47:07.394 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-01-18T06:47:07.394 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-01-18T06:47:07.394 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-01-18T06:47:07.394 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-01-18T06:47:07.394 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-01-18T06:47:07.394 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-01-18T06:47:07.394 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-01-18T06:47:07.394 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-01-18T06:47:07.394 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-01-18T06:47:07.394 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-01-18T06:47:07.395 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-01-18T06:47:07.395 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-01-18T06:47:07.395 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-01-18T06:47:07.395 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-01-18T06:47:07.395 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-01-18T06:47:07.395 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-01-18T06:47:07.395 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-01-18T06:47:07.395 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-01-18T06:47:07.396 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-01-18T06:47:07.396 DEBUG:teuthology.orchestra.run.smithi047:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-01-18T06:47:07.424 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 653e4012-b5cd-11ee-95ae-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-01-18T06:47:07.424 DEBUG:teuthology.orchestra.run.smithi047:mon.smithi047> sudo journalctl -f -n 0 -u ceph-653e4012-b5cd-11ee-95ae-87774f69a715@mon.smithi047.service 2024-01-18T06:47:07.466 INFO:tasks.cephadm:Bootstrapping... 2024-01-18T06:47:07.466 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.47 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-01-18T06:47:07.495 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:-- Logs begin at Thu 2024-01-18 06:37:32 UTC. -- 2024-01-18T06:47:07.736 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-01-18T06:47:08.612 INFO:teuthology.orchestra.run.smithi047.stderr:/bin/podman: 4.6.1 2024-01-18T06:47:08.622 INFO:teuthology.orchestra.run.smithi047.stderr:Verifying podman|docker is present... 2024-01-18T06:47:08.622 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-01-18T06:47:08.646 INFO:teuthology.orchestra.run.smithi047.stderr:/bin/podman: 4.6.1 2024-01-18T06:47:08.658 INFO:teuthology.orchestra.run.smithi047.stderr:Verifying lvm2 is present... 2024-01-18T06:47:08.659 INFO:teuthology.orchestra.run.smithi047.stderr:Verifying time synchronization is in place... 2024-01-18T06:47:08.659 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl is-enabled chrony.service 2024-01-18T06:47:08.667 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-01-18T06:47:08.667 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl is-active chrony.service 2024-01-18T06:47:08.674 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: inactive 2024-01-18T06:47:08.674 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl is-enabled chronyd.service 2024-01-18T06:47:08.681 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: enabled 2024-01-18T06:47:08.682 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl is-active chronyd.service 2024-01-18T06:47:08.687 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: active 2024-01-18T06:47:08.688 INFO:teuthology.orchestra.run.smithi047.stderr:Unit chronyd.service is enabled and running 2024-01-18T06:47:08.688 INFO:teuthology.orchestra.run.smithi047.stderr:Repeating the final host check... 2024-01-18T06:47:08.688 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-01-18T06:47:08.710 INFO:teuthology.orchestra.run.smithi047.stderr:/bin/podman: 4.6.1 2024-01-18T06:47:08.723 INFO:teuthology.orchestra.run.smithi047.stderr:podman|docker (/bin/podman) is present 2024-01-18T06:47:08.723 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl is present 2024-01-18T06:47:08.723 INFO:teuthology.orchestra.run.smithi047.stderr:lvcreate is present 2024-01-18T06:47:08.723 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl is-enabled chrony.service 2024-01-18T06:47:08.729 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-01-18T06:47:08.729 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl is-active chrony.service 2024-01-18T06:47:08.735 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: inactive 2024-01-18T06:47:08.736 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl is-enabled chronyd.service 2024-01-18T06:47:08.742 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: enabled 2024-01-18T06:47:08.742 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl is-active chronyd.service 2024-01-18T06:47:08.748 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: active 2024-01-18T06:47:08.750 INFO:teuthology.orchestra.run.smithi047.stderr:Unit chronyd.service is enabled and running 2024-01-18T06:47:08.750 INFO:teuthology.orchestra.run.smithi047.stderr:Host looks OK 2024-01-18T06:47:08.750 INFO:teuthology.orchestra.run.smithi047.stderr:Cluster fsid: 653e4012-b5cd-11ee-95ae-87774f69a715 2024-01-18T06:47:08.750 INFO:teuthology.orchestra.run.smithi047.stderr:Acquiring lock 140595555518952 on /run/cephadm/653e4012-b5cd-11ee-95ae-87774f69a715.lock 2024-01-18T06:47:08.750 INFO:teuthology.orchestra.run.smithi047.stderr:Lock 140595555518952 acquired on /run/cephadm/653e4012-b5cd-11ee-95ae-87774f69a715.lock 2024-01-18T06:47:08.750 INFO:teuthology.orchestra.run.smithi047.stderr:Verifying IP 172.21.15.47 port 3300 ... 2024-01-18T06:47:08.751 INFO:teuthology.orchestra.run.smithi047.stderr:Verifying IP 172.21.15.47 port 6789 ... 2024-01-18T06:47:08.751 INFO:teuthology.orchestra.run.smithi047.stderr:Base mon IP is 172.21.15.47, final addrv is [v2:172.21.15.47:3300,v1:172.21.15.47:6789] 2024-01-18T06:47:08.751 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /sbin/ip route ls 2024-01-18T06:47:08.753 INFO:teuthology.orchestra.run.smithi047.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.47 metric 100 2024-01-18T06:47:08.753 INFO:teuthology.orchestra.run.smithi047.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.47 metric 100 2024-01-18T06:47:08.754 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /sbin/ip -6 route ls 2024-01-18T06:47:08.756 INFO:teuthology.orchestra.run.smithi047.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-01-18T06:47:08.756 INFO:teuthology.orchestra.run.smithi047.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-01-18T06:47:08.756 INFO:teuthology.orchestra.run.smithi047.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1752sec hoplimit 64 pref medium 2024-01-18T06:47:08.757 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /sbin/ip -6 addr ls 2024-01-18T06:47:08.759 INFO:teuthology.orchestra.run.smithi047.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-01-18T06:47:08.759 INFO:teuthology.orchestra.run.smithi047.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-01-18T06:47:08.759 INFO:teuthology.orchestra.run.smithi047.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-01-18T06:47:08.759 INFO:teuthology.orchestra.run.smithi047.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2024-01-18T06:47:08.759 INFO:teuthology.orchestra.run.smithi047.stderr:/sbin/ip: inet6 fe80::ec4:7aff:febd:148e/64 scope link 2024-01-18T06:47:08.760 INFO:teuthology.orchestra.run.smithi047.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-01-18T06:47:08.760 INFO:teuthology.orchestra.run.smithi047.stderr:Mon IP 172.21.15.47 is in CIDR network 172.21.0.0/20 2024-01-18T06:47:08.760 INFO:teuthology.orchestra.run.smithi047.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-01-18T06:47:08.762 INFO:teuthology.orchestra.run.smithi047.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2024-01-18T06:47:08.762 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2024-01-18T06:47:08.860 INFO:teuthology.orchestra.run.smithi047.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2024-01-18T06:47:15.407 INFO:teuthology.orchestra.run.smithi047.stderr:/bin/podman: Getting image source signatures 2024-01-18T06:47:15.415 INFO:teuthology.orchestra.run.smithi047.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2024-01-18T06:47:15.416 INFO:teuthology.orchestra.run.smithi047.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2024-01-18T06:47:27.666 INFO:teuthology.orchestra.run.smithi047.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-01-18T06:47:27.749 INFO:teuthology.orchestra.run.smithi047.stderr:/bin/podman: Writing manifest to image destination 2024-01-18T06:47:28.112 INFO:teuthology.orchestra.run.smithi047.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-01-18T06:47:28.130 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2024-01-18T06:47:28.704 INFO:teuthology.orchestra.run.smithi047.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-01-18T06:47:29.082 INFO:teuthology.orchestra.run.smithi047.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-01-18T06:47:29.082 INFO:teuthology.orchestra.run.smithi047.stderr:Extracting ceph user uid/gid from container image... 2024-01-18T06:47:29.082 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2024-01-18T06:47:29.385 INFO:teuthology.orchestra.run.smithi047.stderr:stat: 167 167 2024-01-18T06:47:29.722 INFO:teuthology.orchestra.run.smithi047.stderr:Creating initial keys... 2024-01-18T06:47:29.722 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-01-18T06:47:30.017 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-authtool: AQCCyahl/mJJABAAofVcTpdNY2VPmLAbvm5Nqw== 2024-01-18T06:47:30.370 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-01-18T06:47:30.650 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-authtool: AQCCyahlykGGJRAA9vhSUiokSjB7f4IcL066zA== 2024-01-18T06:47:30.976 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-01-18T06:47:31.300 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-authtool: AQCDyahlZwAnERAAGR7PtWsXHcAkqh0f9XXtJg== 2024-01-18T06:47:31.676 INFO:teuthology.orchestra.run.smithi047.stderr:Creating initial monmap... 2024-01-18T06:47:31.676 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmprrpcwb0m:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 --addv smithi047 [v2:172.21.15.47:3300,v1:172.21.15.47:6789] /tmp/monmap 2024-01-18T06:47:31.968 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-18T06:47:31.969 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 653e4012-b5cd-11ee-95ae-87774f69a715 2024-01-18T06:47:31.969 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-18T06:47:32.306 INFO:teuthology.orchestra.run.smithi047.stderr:monmaptool for smithi047 [v2:172.21.15.47:3300,v1:172.21.15.47:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-18T06:47:32.306 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/monmaptool: set fsid to 653e4012-b5cd-11ee-95ae-87774f69a715 2024-01-18T06:47:32.306 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-18T06:47:32.306 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-01-18T06:47:32.306 INFO:teuthology.orchestra.run.smithi047.stderr:Creating mon... 2024-01-18T06:47:32.307 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/653e4012-b5cd-11ee-95ae-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/mon.smithi047:/var/lib/ceph/mon/ceph-smithi047:z -v /tmp/ceph-tmpxo9tm3fw:/tmp/keyring:z -v /tmp/ceph-tmprrpcwb0m:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi047 --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2024-01-18T06:47:32.619 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.585+0000 7f30fd51a700 0 set uid:gid to 167:167 (ceph:ceph) 2024-01-18T06:47:32.619 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 1 imported monmap: 2024-01-18T06:47:32.619 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: epoch 0 2024-01-18T06:47:32.619 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: fsid 653e4012-b5cd-11ee-95ae-87774f69a715 2024-01-18T06:47:32.619 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: last_changed 2024-01-18T06:47:31.951147+0000 2024-01-18T06:47:32.619 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: created 2024-01-18T06:47:31.951147+0000 2024-01-18T06:47:32.619 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2024-01-18T06:47:32.620 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: election_strategy: 1 2024-01-18T06:47:32.620 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.smithi047 2024-01-18T06:47:32.620 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: 2024-01-18T06:47:32.620 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 0 /usr/bin/ceph-mon: set fsid to 653e4012-b5cd-11ee-95ae-87774f69a715 2024-01-18T06:47:32.620 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: RocksDB version: 6.8.1 2024-01-18T06:47:32.620 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: 2024-01-18T06:47:32.620 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-18T06:47:32.620 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Compile date May 13 2021 2024-01-18T06:47:32.620 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: DB SUMMARY 2024-01-18T06:47:32.620 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: 2024-01-18T06:47:32.620 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi047/store.db dir, Total Num: 0, files: 2024-01-18T06:47:32.620 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: 2024-01-18T06:47:32.620 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi047/store.db: 2024-01-18T06:47:32.620 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: 2024-01-18T06:47:32.620 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.error_if_exists: 0 2024-01-18T06:47:32.621 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.create_if_missing: 1 2024-01-18T06:47:32.621 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.paranoid_checks: 1 2024-01-18T06:47:32.621 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.env: 0x55d4018431c0 2024-01-18T06:47:32.621 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.fs: Posix File System 2024-01-18T06:47:32.621 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.info_log: 0x55d4040f6ec0 2024-01-18T06:47:32.621 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.max_file_opening_threads: 16 2024-01-18T06:47:32.621 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.statistics: (nil) 2024-01-18T06:47:32.621 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.use_fsync: 0 2024-01-18T06:47:32.621 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.max_log_file_size: 0 2024-01-18T06:47:32.621 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-18T06:47:32.621 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-01-18T06:47:32.621 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.keep_log_file_num: 1000 2024-01-18T06:47:32.621 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.recycle_log_file_num: 0 2024-01-18T06:47:32.621 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.allow_fallocate: 1 2024-01-18T06:47:32.621 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.allow_mmap_reads: 0 2024-01-18T06:47:32.622 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.allow_mmap_writes: 0 2024-01-18T06:47:32.622 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.use_direct_reads: 0 2024-01-18T06:47:32.622 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-18T06:47:32.622 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.create_missing_column_families: 0 2024-01-18T06:47:32.622 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.db_log_dir: 2024-01-18T06:47:32.622 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi047/store.db 2024-01-18T06:47:32.622 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-01-18T06:47:32.622 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.max_subcompactions: 1 2024-01-18T06:47:32.622 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.max_background_flushes: -1 2024-01-18T06:47:32.622 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-01-18T06:47:32.622 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-01-18T06:47:32.622 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-18T06:47:32.622 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-18T06:47:32.623 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-01-18T06:47:32.623 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.advise_random_on_open: 1 2024-01-18T06:47:32.623 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.db_write_buffer_size: 0 2024-01-18T06:47:32.623 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.write_buffer_manager: 0x55d4040fd530 2024-01-18T06:47:32.623 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-18T06:47:32.623 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-18T06:47:32.623 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-18T06:47:32.623 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-01-18T06:47:32.623 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.rate_limiter: (nil) 2024-01-18T06:47:32.623 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-18T06:47:32.623 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.wal_recovery_mode: 2 2024-01-18T06:47:32.623 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.enable_thread_tracking: 0 2024-01-18T06:47:32.623 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.enable_pipelined_write: 0 2024-01-18T06:47:32.623 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.unordered_write: 0 2024-01-18T06:47:32.623 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-18T06:47:32.624 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-18T06:47:32.624 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-18T06:47:32.624 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-18T06:47:32.624 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.row_cache: None 2024-01-18T06:47:32.624 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.wal_filter: None 2024-01-18T06:47:32.624 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-18T06:47:32.624 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.allow_ingest_behind: 0 2024-01-18T06:47:32.624 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.preserve_deletes: 0 2024-01-18T06:47:32.624 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.two_write_queues: 0 2024-01-18T06:47:32.624 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.manual_wal_flush: 0 2024-01-18T06:47:32.624 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.atomic_flush: 0 2024-01-18T06:47:32.627 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-18T06:47:32.627 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-01-18T06:47:32.627 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-01-18T06:47:32.627 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.log_readahead_size: 0 2024-01-18T06:47:32.627 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-01-18T06:47:32.627 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.max_background_jobs: 2 2024-01-18T06:47:32.627 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.max_background_compactions: -1 2024-01-18T06:47:32.627 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-18T06:47:32.627 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-18T06:47:32.627 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-01-18T06:47:32.627 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.max_total_wal_size: 0 2024-01-18T06:47:32.627 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-18T06:47:32.627 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-01-18T06:47:32.628 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-01-18T06:47:32.628 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-18T06:47:32.628 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.max_open_files: -1 2024-01-18T06:47:32.628 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.bytes_per_sync: 0 2024-01-18T06:47:32.628 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-01-18T06:47:32.628 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-01-18T06:47:32.628 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Options.compaction_readahead_size: 0 2024-01-18T06:47:32.628 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Compression algorithms supported: 2024-01-18T06:47:32.628 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-18T06:47:32.628 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: kZSTD supported: 0 2024-01-18T06:47:32.628 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: kXpressCompression supported: 0 2024-01-18T06:47:32.628 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: kLZ4HCCompression supported: 1 2024-01-18T06:47:32.628 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: kLZ4Compression supported: 1 2024-01-18T06:47:32.629 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: kBZip2Compression supported: 0 2024-01-18T06:47:32.629 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: kZlibCompression supported: 1 2024-01-18T06:47:32.629 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: kSnappyCompression supported: 1 2024-01-18T06:47:32.629 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.586+0000 7f30fd51a700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-01-18T06:47:32.629 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2024-01-18T06:47:32.629 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: 2024-01-18T06:47:32.629 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi047/store.db/MANIFEST-000001 2024-01-18T06:47:32.629 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: 2024-01-18T06:47:32.629 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-01-18T06:47:32.629 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: 2024-01-18T06:47:32.629 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-18T06:47:32.629 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.merge_operator: 2024-01-18T06:47:32.630 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.compaction_filter: None 2024-01-18T06:47:32.630 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.compaction_filter_factory: None 2024-01-18T06:47:32.630 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-01-18T06:47:32.630 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-01-18T06:47:32.630 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d40400e120) 2024-01-18T06:47:32.630 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2024-01-18T06:47:32.630 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-18T06:47:32.630 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-18T06:47:32.630 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2024-01-18T06:47:32.630 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: index_type: 0 2024-01-18T06:47:32.630 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2024-01-18T06:47:32.630 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: index_shortening: 1 2024-01-18T06:47:32.630 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2024-01-18T06:47:32.630 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2024-01-18T06:47:32.630 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: checksum: 1 2024-01-18T06:47:32.631 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2024-01-18T06:47:32.631 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: block_cache: 0x55d404044d10 2024-01-18T06:47:32.631 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2024-01-18T06:47:32.631 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: block_cache_options: 2024-01-18T06:47:32.631 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: capacity : 536870912 2024-01-18T06:47:32.631 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2024-01-18T06:47:32.631 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2024-01-18T06:47:32.631 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2024-01-18T06:47:32.631 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2024-01-18T06:47:32.631 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2024-01-18T06:47:32.631 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: block_size: 4096 2024-01-18T06:47:32.631 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2024-01-18T06:47:32.631 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2024-01-18T06:47:32.631 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2024-01-18T06:47:32.631 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2024-01-18T06:47:32.632 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: partition_filters: 0 2024-01-18T06:47:32.632 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2024-01-18T06:47:32.632 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-18T06:47:32.632 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2024-01-18T06:47:32.632 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: verify_compression: 0 2024-01-18T06:47:32.632 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2024-01-18T06:47:32.632 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: format_version: 2 2024-01-18T06:47:32.632 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2024-01-18T06:47:32.632 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: block_align: 0 2024-01-18T06:47:32.632 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: 2024-01-18T06:47:32.632 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.write_buffer_size: 33554432 2024-01-18T06:47:32.632 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.max_write_buffer_number: 2 2024-01-18T06:47:32.633 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.compression: NoCompression 2024-01-18T06:47:32.633 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.bottommost_compression: Disabled 2024-01-18T06:47:32.633 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.prefix_extractor: nullptr 2024-01-18T06:47:32.633 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-18T06:47:32.633 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.num_levels: 7 2024-01-18T06:47:32.633 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-18T06:47:32.633 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-18T06:47:32.633 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-18T06:47:32.633 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-18T06:47:32.633 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-18T06:47:32.633 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-18T06:47:32.633 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-18T06:47:32.633 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-18T06:47:32.633 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-18T06:47:32.634 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-01-18T06:47:32.634 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.compression_opts.level: 32767 2024-01-18T06:47:32.634 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.compression_opts.strategy: 0 2024-01-18T06:47:32.634 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-18T06:47:32.634 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-18T06:47:32.634 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.compression_opts.enabled: false 2024-01-18T06:47:32.634 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-18T06:47:32.634 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-18T06:47:32.634 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-18T06:47:32.634 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.target_file_size_base: 67108864 2024-01-18T06:47:32.634 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-01-18T06:47:32.634 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-18T06:47:32.634 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-18T06:47:32.634 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-18T06:47:32.635 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-18T06:47:32.635 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-18T06:47:32.635 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-18T06:47:32.635 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-18T06:47:32.635 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-18T06:47:32.635 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-18T06:47:32.635 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-18T06:47:32.635 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-18T06:47:32.636 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-18T06:47:32.636 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.arena_block_size: 4194304 2024-01-18T06:47:32.636 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-18T06:47:32.636 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-18T06:47:32.636 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-18T06:47:32.636 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.disable_auto_compactions: 0 2024-01-18T06:47:32.637 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-18T06:47:32.637 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-18T06:47:32.637 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-18T06:47:32.637 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-18T06:47:32.637 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-18T06:47:32.638 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-18T06:47:32.638 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-18T06:47:32.638 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-18T06:47:32.638 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-18T06:47:32.638 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-18T06:47:32.638 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.table_properties_collectors: 2024-01-18T06:47:32.639 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.inplace_update_support: 0 2024-01-18T06:47:32.639 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-01-18T06:47:32.639 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-18T06:47:32.639 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-18T06:47:32.639 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-01-18T06:47:32.639 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.bloom_locality: 0 2024-01-18T06:47:32.639 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.max_successive_merges: 0 2024-01-18T06:47:32.640 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-01-18T06:47:32.640 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.paranoid_file_checks: 0 2024-01-18T06:47:32.640 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.force_consistency_checks: 0 2024-01-18T06:47:32.640 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.report_bg_io_stats: 0 2024-01-18T06:47:32.640 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.ttl: 2592000 2024-01-18T06:47:32.640 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.587+0000 7f30fd51a700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-01-18T06:47:32.641 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.588+0000 7f30fd51a700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi047/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-18T06:47:32.641 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: 2024-01-18T06:47:32.641 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.588+0000 7f30fd51a700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-01-18T06:47:32.641 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: 2024-01-18T06:47:32.641 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.588+0000 7f30fd51a700 4 rocksdb: DB pointer 0x55d404109800 2024-01-18T06:47:32.641 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.588+0000 7f30e63b3700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-01-18T06:47:32.642 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.588+0000 7f30e63b3700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-01-18T06:47:32.642 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2024-01-18T06:47:32.642 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-18T06:47:32.642 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-18T06:47:32.642 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-18T06:47:32.642 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-18T06:47:32.643 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-18T06:47:32.643 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-18T06:47:32.643 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-18T06:47:32.643 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: 2024-01-18T06:47:32.643 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-18T06:47:32.643 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-18T06:47:32.643 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-18T06:47:32.644 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-18T06:47:32.644 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-18T06:47:32.644 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: 2024-01-18T06:47:32.644 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-18T06:47:32.644 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-18T06:47:32.644 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-18T06:47:32.645 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-18T06:47:32.645 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-01-18T06:47:32.645 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-18T06:47:32.645 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-01-18T06:47:32.645 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-01-18T06:47:32.645 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-01-18T06:47:32.645 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-18T06:47:32.646 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-18T06:47:32.646 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-18T06:47:32.646 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: 2024-01-18T06:47:32.646 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-01-18T06:47:32.646 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: 2024-01-18T06:47:32.646 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-18T06:47:32.647 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-18T06:47:32.647 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-18T06:47:32.647 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-18T06:47:32.647 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-18T06:47:32.647 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: 2024-01-18T06:47:32.647 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-18T06:47:32.647 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-18T06:47:32.648 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-18T06:47:32.648 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-18T06:47:32.648 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-01-18T06:47:32.648 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-18T06:47:32.648 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-01-18T06:47:32.648 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-01-18T06:47:32.649 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-01-18T06:47:32.649 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-18T06:47:32.649 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-18T06:47:32.649 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-18T06:47:32.649 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: 2024-01-18T06:47:32.649 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-01-18T06:47:32.649 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: 2024-01-18T06:47:32.650 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.590+0000 7f30fd51a700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2024-01-18T06:47:32.650 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.590+0000 7f30fd51a700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2024-01-18T06:47:32.650 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph-mon: debug 2024-01-18T06:47:32.590+0000 7f30fd51a700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi047 for mon.smithi047 2024-01-18T06:47:32.941 INFO:teuthology.orchestra.run.smithi047.stderr:create mon.smithi047 on 2024-01-18T06:47:32.941 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/653e4012-b5cd-11ee-95ae-87774f69a715 2024-01-18T06:47:32.946 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl disable ceph.target 2024-01-18T06:47:32.953 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-01-18T06:47:33.040 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl enable ceph.target 2024-01-18T06:47:33.047 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-01-18T06:47:33.132 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl start ceph.target 2024-01-18T06:47:33.139 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl enable ceph-653e4012-b5cd-11ee-95ae-87774f69a715.target 2024-01-18T06:47:33.145 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-653e4012-b5cd-11ee-95ae-87774f69a715.target → /etc/systemd/system/ceph-653e4012-b5cd-11ee-95ae-87774f69a715.target. 2024-01-18T06:47:33.146 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-653e4012-b5cd-11ee-95ae-87774f69a715.target → /etc/systemd/system/ceph-653e4012-b5cd-11ee-95ae-87774f69a715.target. 2024-01-18T06:47:33.231 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl start ceph-653e4012-b5cd-11ee-95ae-87774f69a715.target 2024-01-18T06:47:33.240 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl daemon-reload 2024-01-18T06:47:33.326 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl stop ceph-653e4012-b5cd-11ee-95ae-87774f69a715@mon.smithi047 2024-01-18T06:47:33.352 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl reset-failed ceph-653e4012-b5cd-11ee-95ae-87774f69a715@mon.smithi047 2024-01-18T06:47:33.358 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: Failed to reset failed state of unit ceph-653e4012-b5cd-11ee-95ae-87774f69a715@mon.smithi047.service: Unit ceph-653e4012-b5cd-11ee-95ae-87774f69a715@mon.smithi047.service not loaded. 2024-01-18T06:47:33.358 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl enable ceph-653e4012-b5cd-11ee-95ae-87774f69a715@mon.smithi047 2024-01-18T06:47:33.365 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: Created symlink /etc/systemd/system/ceph-653e4012-b5cd-11ee-95ae-87774f69a715.target.wants/ceph-653e4012-b5cd-11ee-95ae-87774f69a715@mon.smithi047.service → /etc/systemd/system/ceph-653e4012-b5cd-11ee-95ae-87774f69a715@.service. 2024-01-18T06:47:33.454 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl start ceph-653e4012-b5cd-11ee-95ae-87774f69a715@mon.smithi047 2024-01-18T06:47:33.871 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-18T06:47:33.880 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: disabled 2024-01-18T06:47:33.880 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl is-active firewalld.service 2024-01-18T06:47:33.888 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: inactive 2024-01-18T06:47:33.888 INFO:teuthology.orchestra.run.smithi047.stderr:firewalld.service is not enabled 2024-01-18T06:47:33.888 INFO:teuthology.orchestra.run.smithi047.stderr:Not possible to enable service . firewalld.service is not available 2024-01-18T06:47:33.888 INFO:teuthology.orchestra.run.smithi047.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-01-18T06:47:33.889 INFO:teuthology.orchestra.run.smithi047.stderr:Waiting for mon to start... 2024-01-18T06:47:33.889 INFO:teuthology.orchestra.run.smithi047.stderr:Waiting for mon... 2024-01-18T06:47:33.889 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/mon.smithi047:/var/lib/ceph/mon/ceph-smithi047:z -v /tmp/ceph-tmpck5s1jz7:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcg0aieo2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2024-01-18T06:47:34.202 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:33 smithi047 systemd[1]: Started Ceph mon.smithi047 for 653e4012-b5cd-11ee-95ae-87774f69a715. 2024-01-18T06:47:34.202 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:33 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38390]: cluster 2024-01-18T06:47:33.869290+0000 mon.smithi047 (mon.0) 0 : [INF] mkfs 653e4012-b5cd-11ee-95ae-87774f69a715 2024-01-18T06:47:34.202 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:33 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38390]: cluster 2024-01-18T06:47:33.854331+0000 mon.smithi047 (mon.0) 1 : cluster [INF] mon.smithi047 is new leader, mons smithi047 in quorum (ranks 0) 2024-01-18T06:47:34.472 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.470+0000 7fa0b2a18700 1 Processor -- start 2024-01-18T06:47:34.472 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.470+0000 7fa0b2a18700 1 -- start start 2024-01-18T06:47:34.473 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.471+0000 7fa0b2a18700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0ac0703b0 0x7fa0ac100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T06:47:34.473 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.471+0000 7fa0b2a18700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0ac0fc5a0 con 0x7fa0ac0703b0 2024-01-18T06:47:34.473 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.471+0000 7fa0b1a16700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0ac0703b0 0x7fa0ac100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:47:34.473 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.471+0000 7fa0b1a16700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0ac0703b0 0x7fa0ac100bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:42746/0 (socket says 172.21.15.47:42746) 2024-01-18T06:47:34.473 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.471+0000 7fa0b1a16700 1 -- 172.21.15.47:0/2763181856 learned_addr learned my addr 172.21.15.47:0/2763181856 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:47:34.473 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.471+0000 7fa0b1a16700 1 -- 172.21.15.47:0/2763181856 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0ac1010a0 con 0x7fa0ac0703b0 2024-01-18T06:47:34.473 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.472+0000 7fa0b1a16700 1 --2- 172.21.15.47:0/2763181856 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0ac0703b0 0x7fa0ac100bb0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fa0a800b5f0 tx=0x7fa0a8006ee0).ready entity=mon.0 client_cookie=4ddfc5d004ed087 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:47:34.476 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.472+0000 7fa0b0a14700 1 -- 172.21.15.47:0/2763181856 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0a800d600 con 0x7fa0ac0703b0 2024-01-18T06:47:34.476 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.472+0000 7fa0b0a14700 1 -- 172.21.15.47:0/2763181856 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0a800d760 con 0x7fa0ac0703b0 2024-01-18T06:47:34.478 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.472+0000 7fa0b0a14700 1 -- 172.21.15.47:0/2763181856 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0a800da30 con 0x7fa0ac0703b0 2024-01-18T06:47:34.479 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.472+0000 7fa0b2a18700 1 -- 172.21.15.47:0/2763181856 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0ac0703b0 msgr2=0x7fa0ac100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:47:34.479 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.472+0000 7fa0b2a18700 1 --2- 172.21.15.47:0/2763181856 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0ac0703b0 0x7fa0ac100bb0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fa0a800b5f0 tx=0x7fa0a8006ee0).stop 2024-01-18T06:47:34.479 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.473+0000 7fa0b2a18700 1 -- 172.21.15.47:0/2763181856 shutdown_connections 2024-01-18T06:47:34.479 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.473+0000 7fa0b2a18700 1 --2- 172.21.15.47:0/2763181856 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0ac0703b0 0x7fa0ac100bb0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:47:34.480 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.473+0000 7fa0b2a18700 1 -- 172.21.15.47:0/2763181856 >> 172.21.15.47:0/2763181856 conn(0x7fa0ac0f6210 msgr2=0x7fa0ac0f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:47:34.480 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.473+0000 7fa0b2a18700 1 -- 172.21.15.47:0/2763181856 shutdown_connections 2024-01-18T06:47:34.480 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.473+0000 7fa0b2a18700 1 -- 172.21.15.47:0/2763181856 wait complete. 2024-01-18T06:47:34.480 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.473+0000 7fa0b2a18700 1 Processor -- start 2024-01-18T06:47:34.480 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.473+0000 7fa0b2a18700 1 -- start start 2024-01-18T06:47:34.480 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.473+0000 7fa0b2a18700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0ac0703b0 0x7fa0ac10c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:47:34.480 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.473+0000 7fa0b2a18700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0ac10c7d0 con 0x7fa0ac0703b0 2024-01-18T06:47:34.480 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.474+0000 7fa0b1a16700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0ac0703b0 0x7fa0ac10c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:47:34.480 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.474+0000 7fa0b1a16700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0ac0703b0 0x7fa0ac10c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:42760/0 (socket says 172.21.15.47:42760) 2024-01-18T06:47:34.480 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.474+0000 7fa0b1a16700 1 -- 172.21.15.47:0/2833677560 learned_addr learned my addr 172.21.15.47:0/2833677560 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:47:34.480 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.474+0000 7fa0b1a16700 1 -- 172.21.15.47:0/2833677560 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a800b200 con 0x7fa0ac0703b0 2024-01-18T06:47:34.480 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.474+0000 7fa0b1a16700 1 --2- 172.21.15.47:0/2833677560 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0ac0703b0 0x7fa0ac10c2e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fa0a8006c70 tx=0x7fa0a8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:47:34.480 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.474+0000 7fa09a7fc700 1 -- 172.21.15.47:0/2833677560 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0a8011b40 con 0x7fa0ac0703b0 2024-01-18T06:47:34.481 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.474+0000 7fa09a7fc700 1 -- 172.21.15.47:0/2833677560 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0a8011ca0 con 0x7fa0ac0703b0 2024-01-18T06:47:34.481 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.474+0000 7fa0b2a18700 1 -- 172.21.15.47:0/2833677560 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0ac10c9d0 con 0x7fa0ac0703b0 2024-01-18T06:47:34.481 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.474+0000 7fa09a7fc700 1 -- 172.21.15.47:0/2833677560 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0a801d430 con 0x7fa0ac0703b0 2024-01-18T06:47:34.481 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.474+0000 7fa0b2a18700 1 -- 172.21.15.47:0/2833677560 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0ac10cde0 con 0x7fa0ac0703b0 2024-01-18T06:47:34.481 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.474+0000 7fa09a7fc700 1 -- 172.21.15.47:0/2833677560 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fa0a800f070 con 0x7fa0ac0703b0 2024-01-18T06:47:34.481 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.474+0000 7fa09a7fc700 1 -- 172.21.15.47:0/2833677560 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fa0a8027460 con 0x7fa0ac0703b0 2024-01-18T06:47:34.481 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.475+0000 7fa0b2a18700 1 -- 172.21.15.47:0/2833677560 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0ac045ad0 con 0x7fa0ac0703b0 2024-01-18T06:47:34.481 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.478+0000 7fa09a7fc700 1 -- 172.21.15.47:0/2833677560 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fa0a8016070 con 0x7fa0ac0703b0 2024-01-18T06:47:34.550 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.547+0000 7fa0b2a18700 1 -- 172.21.15.47:0/2833677560 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fa0ac105de0 con 0x7fa0ac0703b0 2024-01-18T06:47:34.550 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.547+0000 7fa09a7fc700 1 -- 172.21.15.47:0/2833677560 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7fa0a8036070 con 0x7fa0ac0703b0 2024-01-18T06:47:34.550 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: cluster: 2024-01-18T06:47:34.550 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: id: 653e4012-b5cd-11ee-95ae-87774f69a715 2024-01-18T06:47:34.550 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-01-18T06:47:34.550 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.550 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: services: 2024-01-18T06:47:34.550 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi047 (age 0.680417s) 2024-01-18T06:47:34.550 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: mgr: no daemons active 2024-01-18T06:47:34.550 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-01-18T06:47:34.550 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.550 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: data: 2024-01-18T06:47:34.551 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-01-18T06:47:34.551 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-01-18T06:47:34.551 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-01-18T06:47:34.551 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: pgs: 2024-01-18T06:47:34.551 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.551 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.548+0000 7fa0b2a18700 1 -- 172.21.15.47:0/2833677560 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0ac0703b0 msgr2=0x7fa0ac10c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:47:34.551 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.548+0000 7fa0b2a18700 1 --2- 172.21.15.47:0/2833677560 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0ac0703b0 0x7fa0ac10c2e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fa0a8006c70 tx=0x7fa0a8016000).stop 2024-01-18T06:47:34.551 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.548+0000 7fa0b2a18700 1 -- 172.21.15.47:0/2833677560 shutdown_connections 2024-01-18T06:47:34.551 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.548+0000 7fa0b2a18700 1 --2- 172.21.15.47:0/2833677560 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0ac0703b0 0x7fa0ac10c2e0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:47:34.551 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.548+0000 7fa0b2a18700 1 -- 172.21.15.47:0/2833677560 >> 172.21.15.47:0/2833677560 conn(0x7fa0ac0f6210 msgr2=0x7fa0ac0f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:47:34.551 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.548+0000 7fa0b2a18700 1 -- 172.21.15.47:0/2833677560 shutdown_connections 2024-01-18T06:47:34.551 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:34.548+0000 7fa0b2a18700 1 -- 172.21.15.47:0/2833677560 wait complete. 2024-01-18T06:47:34.934 INFO:teuthology.orchestra.run.smithi047.stderr:mon is available 2024-01-18T06:47:34.934 INFO:teuthology.orchestra.run.smithi047.stderr:Assimilating anything we can from ceph.conf... 2024-01-18T06:47:34.934 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/653e4012-b5cd-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpck5s1jz7:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcg0aieo2:/etc/ceph/ceph.conf:z -v /var/lib/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/mon.smithi047:/var/lib/ceph/mon/ceph-smithi047:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi047/config 2024-01-18T06:47:35.145 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:34 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38390]: cluster 2024-01-18T06:47:33.868846+0000 mon.smithi047 (mon.0) 2 : cluster [INF] mon.smithi047 is new leader, mons smithi047 in quorum (ranks 0) 2024-01-18T06:47:35.146 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:34 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38390]: cluster 2024-01-18T06:47:33.868971+0000 mon.smithi047 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi047=[v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0]} 2024-01-18T06:47:35.146 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:34 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38390]: cluster 2024-01-18T06:47:33.870537+0000 mon.smithi047 (mon.0) 4 : cluster [DBG] fsmap 2024-01-18T06:47:35.146 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:34 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38390]: cluster 2024-01-18T06:47:33.872518+0000 mon.smithi047 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-01-18T06:47:35.146 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:34 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38390]: cluster 2024-01-18T06:47:33.872895+0000 mon.smithi047 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-01-18T06:47:35.146 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:34 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38390]: audit 2024-01-18T06:47:34.549233+0000 mon.smithi047 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.47:0/2833677560' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-01-18T06:47:35.435 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.434+0000 7f66a212e700 1 Processor -- start 2024-01-18T06:47:35.439 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.434+0000 7f66a212e700 1 -- start start 2024-01-18T06:47:35.439 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.434+0000 7f66a212e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f669c070430 0x7f669c0fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T06:47:35.439 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.434+0000 7f66a212e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f669c0fedd0 con 0x7f669c070430 2024-01-18T06:47:35.439 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.434+0000 7f669b7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f669c070430 0x7f669c0fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:47:35.439 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.435+0000 7f669b7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f669c070430 0x7f669c0fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:42768/0 (socket says 172.21.15.47:42768) 2024-01-18T06:47:35.439 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.435+0000 7f669b7fe700 1 -- 172.21.15.47:0/3287099000 learned_addr learned my addr 172.21.15.47:0/3287099000 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:47:35.439 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.435+0000 7f669b7fe700 1 -- 172.21.15.47:0/3287099000 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f669c0fef10 con 0x7f669c070430 2024-01-18T06:47:35.439 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.435+0000 7f669b7fe700 1 --2- 172.21.15.47:0/3287099000 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f669c070430 0x7f669c0fe8e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f668c00b5f0 tx=0x7f668c006ee0).ready entity=mon.0 client_cookie=4a99c5a440976086 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:47:35.439 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.435+0000 7f669a7fc700 1 -- 172.21.15.47:0/3287099000 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f668c00d600 con 0x7f669c070430 2024-01-18T06:47:35.439 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.435+0000 7f669a7fc700 1 -- 172.21.15.47:0/3287099000 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f668c00d760 con 0x7f669c070430 2024-01-18T06:47:35.439 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.435+0000 7f669a7fc700 1 -- 172.21.15.47:0/3287099000 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f668c00da30 con 0x7f669c070430 2024-01-18T06:47:35.440 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.436+0000 7f66a212e700 1 -- 172.21.15.47:0/3287099000 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f669c070430 msgr2=0x7f669c0fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:47:35.440 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.436+0000 7f66a212e700 1 --2- 172.21.15.47:0/3287099000 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f669c070430 0x7f669c0fe8e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f668c00b5f0 tx=0x7f668c006ee0).stop 2024-01-18T06:47:35.440 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.436+0000 7f66a212e700 1 -- 172.21.15.47:0/3287099000 shutdown_connections 2024-01-18T06:47:35.440 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.436+0000 7f66a212e700 1 --2- 172.21.15.47:0/3287099000 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f669c070430 0x7f669c0fe8e0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:47:35.440 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.436+0000 7f66a212e700 1 -- 172.21.15.47:0/3287099000 >> 172.21.15.47:0/3287099000 conn(0x7f669c0f63c0 msgr2=0x7f669c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:47:35.440 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.436+0000 7f66a212e700 1 -- 172.21.15.47:0/3287099000 shutdown_connections 2024-01-18T06:47:35.440 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.436+0000 7f66a212e700 1 -- 172.21.15.47:0/3287099000 wait complete. 2024-01-18T06:47:35.440 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.436+0000 7f66a212e700 1 Processor -- start 2024-01-18T06:47:35.440 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.436+0000 7f66a212e700 1 -- start start 2024-01-18T06:47:35.440 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.437+0000 7f66a212e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f669c070430 0x7f669c10a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:47:35.440 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.437+0000 7f66a212e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f669c10a770 con 0x7f669c070430 2024-01-18T06:47:35.440 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.437+0000 7f669b7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f669c070430 0x7f669c10a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:47:35.440 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.437+0000 7f669b7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f669c070430 0x7f669c10a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:42774/0 (socket says 172.21.15.47:42774) 2024-01-18T06:47:35.440 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.437+0000 7f669b7fe700 1 -- 172.21.15.47:0/4111803396 learned_addr learned my addr 172.21.15.47:0/4111803396 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:47:35.441 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.437+0000 7f669b7fe700 1 -- 172.21.15.47:0/4111803396 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f668c00b200 con 0x7f669c070430 2024-01-18T06:47:35.441 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.437+0000 7f669b7fe700 1 --2- 172.21.15.47:0/4111803396 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f669c070430 0x7f669c10a280 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f668c006c70 tx=0x7f668c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:47:35.441 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.437+0000 7f6693fff700 1 -- 172.21.15.47:0/4111803396 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f668c011b40 con 0x7f669c070430 2024-01-18T06:47:35.441 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.437+0000 7f6693fff700 1 -- 172.21.15.47:0/4111803396 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f668c011ca0 con 0x7f669c070430 2024-01-18T06:47:35.441 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.437+0000 7f66a212e700 1 -- 172.21.15.47:0/4111803396 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f669c10a970 con 0x7f669c070430 2024-01-18T06:47:35.441 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.437+0000 7f6693fff700 1 -- 172.21.15.47:0/4111803396 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f668c01d430 con 0x7f669c070430 2024-01-18T06:47:35.441 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.437+0000 7f66a212e700 1 -- 172.21.15.47:0/4111803396 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f669c10ad80 con 0x7f669c070430 2024-01-18T06:47:35.441 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.438+0000 7f6693fff700 1 -- 172.21.15.47:0/4111803396 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f668c00f070 con 0x7f669c070430 2024-01-18T06:47:35.441 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.438+0000 7f6693fff700 1 -- 172.21.15.47:0/4111803396 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f668c027460 con 0x7f669c070430 2024-01-18T06:47:35.441 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.439+0000 7f66a212e700 1 -- 172.21.15.47:0/4111803396 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f669c045ad0 con 0x7f669c070430 2024-01-18T06:47:35.442 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.441+0000 7f6693fff700 1 -- 172.21.15.47:0/4111803396 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f668c016070 con 0x7f669c070430 2024-01-18T06:47:35.508 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.508+0000 7f66a212e700 1 -- 172.21.15.47:0/4111803396 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f669c0ff160 con 0x7f669c070430 2024-01-18T06:47:35.515 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.511+0000 7f6693fff700 1 -- 172.21.15.47:0/4111803396 <== mon.0 v2:172.21.15.47:3300/0 7 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f668c01d5e0 con 0x7f669c070430 2024-01-18T06:47:35.515 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.511+0000 7f6693fff700 1 -- 172.21.15.47:0/4111803396 <== mon.0 v2:172.21.15.47:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+464 (secure 0 0 0) 0x7f668c036020 con 0x7f669c070430 2024-01-18T06:47:35.515 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.515 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: [global] 2024-01-18T06:47:35.515 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: fsid = 653e4012-b5cd-11ee-95ae-87774f69a715 2024-01-18T06:47:35.515 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.47:3300,v1:172.21.15.47:6789] 2024-01-18T06:47:35.515 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-01-18T06:47:35.515 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-01-18T06:47:35.516 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-01-18T06:47:35.516 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-01-18T06:47:35.516 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.516 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: [mgr] 2024-01-18T06:47:35.516 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-01-18T06:47:35.516 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.516 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: [osd] 2024-01-18T06:47:35.516 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2024-01-18T06:47:35.516 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2024-01-18T06:47:35.516 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2024-01-18T06:47:35.516 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-01-18T06:47:35.517 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2024-01-18T06:47:35.517 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-01-18T06:47:35.517 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.512+0000 7f66a212e700 1 -- 172.21.15.47:0/4111803396 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f669c070430 msgr2=0x7f669c10a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:47:35.517 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.512+0000 7f66a212e700 1 --2- 172.21.15.47:0/4111803396 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f669c070430 0x7f669c10a280 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f668c006c70 tx=0x7f668c016000).stop 2024-01-18T06:47:35.517 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.512+0000 7f66a212e700 1 -- 172.21.15.47:0/4111803396 shutdown_connections 2024-01-18T06:47:35.517 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.512+0000 7f66a212e700 1 --2- 172.21.15.47:0/4111803396 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f669c070430 0x7f669c10a280 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:47:35.517 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.512+0000 7f66a212e700 1 -- 172.21.15.47:0/4111803396 >> 172.21.15.47:0/4111803396 conn(0x7f669c0f63c0 msgr2=0x7f669c0f8120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:47:35.517 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.512+0000 7f66a212e700 1 -- 172.21.15.47:0/4111803396 shutdown_connections 2024-01-18T06:47:35.517 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:35.512+0000 7f66a212e700 1 -- 172.21.15.47:0/4111803396 wait complete. 2024-01-18T06:47:35.860 INFO:teuthology.orchestra.run.smithi047.stderr:Generating new minimal ceph.conf... 2024-01-18T06:47:35.860 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/653e4012-b5cd-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpck5s1jz7:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcg0aieo2:/etc/ceph/ceph.conf:z -v /var/lib/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/mon.smithi047:/var/lib/ceph/mon/ceph-smithi047:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi047/config 2024-01-18T06:47:36.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:35 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38390]: audit 2024-01-18T06:47:35.510067+0000 mon.smithi047 (mon.0) 8 : audit [INF] from='client.? 172.21.15.47:0/4111803396' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-18T06:47:36.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:35 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38390]: audit 2024-01-18T06:47:35.513128+0000 mon.smithi047 (mon.0) 9 : audit [INF] from='client.? 172.21.15.47:0/4111803396' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-01-18T06:47:36.233 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.232+0000 7f1ecee8b700 1 Processor -- start 2024-01-18T06:47:36.235 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.232+0000 7f1ecee8b700 1 -- start start 2024-01-18T06:47:36.235 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.232+0000 7f1ecee8b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ec80724d0 0x7f1ec8070a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T06:47:36.235 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.232+0000 7f1ecee8b700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec80728f0 con 0x7f1ec80724d0 2024-01-18T06:47:36.235 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.233+0000 7f1eccc27700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ec80724d0 0x7f1ec8070a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:47:36.235 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.233+0000 7f1eccc27700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ec80724d0 0x7f1ec8070a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:42776/0 (socket says 172.21.15.47:42776) 2024-01-18T06:47:36.235 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.233+0000 7f1eccc27700 1 -- 172.21.15.47:0/30689674 learned_addr learned my addr 172.21.15.47:0/30689674 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:47:36.235 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.233+0000 7f1eccc27700 1 -- 172.21.15.47:0/30689674 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec8070f60 con 0x7f1ec80724d0 2024-01-18T06:47:36.235 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.233+0000 7f1eccc27700 1 --2- 172.21.15.47:0/30689674 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ec80724d0 0x7f1ec8070a70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f1ebc00b5f0 tx=0x7f1ebc006ee0).ready entity=mon.0 client_cookie=dc6ccc84641e5e64 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:47:36.235 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.233+0000 7f1ec77fe700 1 -- 172.21.15.47:0/30689674 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ebc00d600 con 0x7f1ec80724d0 2024-01-18T06:47:36.236 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.233+0000 7f1ec77fe700 1 -- 172.21.15.47:0/30689674 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1ebc00dbd0 con 0x7f1ec80724d0 2024-01-18T06:47:36.236 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.233+0000 7f1ec77fe700 1 -- 172.21.15.47:0/30689674 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ebc011c00 con 0x7f1ec80724d0 2024-01-18T06:47:36.236 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.234+0000 7f1ecee8b700 1 -- 172.21.15.47:0/30689674 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ec80724d0 msgr2=0x7f1ec8070a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:47:36.236 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.234+0000 7f1ecee8b700 1 --2- 172.21.15.47:0/30689674 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ec80724d0 0x7f1ec8070a70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f1ebc00b5f0 tx=0x7f1ebc006ee0).stop 2024-01-18T06:47:36.236 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.234+0000 7f1ecee8b700 1 -- 172.21.15.47:0/30689674 shutdown_connections 2024-01-18T06:47:36.236 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.234+0000 7f1ecee8b700 1 --2- 172.21.15.47:0/30689674 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ec80724d0 0x7f1ec8070a70 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:47:36.236 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.234+0000 7f1ecee8b700 1 -- 172.21.15.47:0/30689674 >> 172.21.15.47:0/30689674 conn(0x7f1ec80f74c0 msgr2=0x7f1ec80f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:47:36.236 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.234+0000 7f1ecee8b700 1 -- 172.21.15.47:0/30689674 shutdown_connections 2024-01-18T06:47:36.236 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.234+0000 7f1ecee8b700 1 -- 172.21.15.47:0/30689674 wait complete. 2024-01-18T06:47:36.236 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.235+0000 7f1ecee8b700 1 Processor -- start 2024-01-18T06:47:36.236 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.235+0000 7f1ecee8b700 1 -- start start 2024-01-18T06:47:36.237 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.235+0000 7f1ecee8b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ec80724d0 0x7f1ec810cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:47:36.237 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.235+0000 7f1ecee8b700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec810d1d0 con 0x7f1ec80724d0 2024-01-18T06:47:36.237 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.235+0000 7f1eccc27700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ec80724d0 0x7f1ec810cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:47:36.238 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.235+0000 7f1eccc27700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ec80724d0 0x7f1ec810cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:42782/0 (socket says 172.21.15.47:42782) 2024-01-18T06:47:36.238 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.235+0000 7f1eccc27700 1 -- 172.21.15.47:0/3124110502 learned_addr learned my addr 172.21.15.47:0/3124110502 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:47:36.238 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.235+0000 7f1eccc27700 1 -- 172.21.15.47:0/3124110502 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ebc00b200 con 0x7f1ec80724d0 2024-01-18T06:47:36.238 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.236+0000 7f1eccc27700 1 --2- 172.21.15.47:0/3124110502 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ec80724d0 0x7f1ec810cce0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f1ebc011820 tx=0x7f1ebc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:47:36.238 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.236+0000 7f1ec57fa700 1 -- 172.21.15.47:0/3124110502 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ebc01b410 con 0x7f1ec80724d0 2024-01-18T06:47:36.238 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.236+0000 7f1ec57fa700 1 -- 172.21.15.47:0/3124110502 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1ebc01b9e0 con 0x7f1ec80724d0 2024-01-18T06:47:36.238 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.236+0000 7f1ec57fa700 1 -- 172.21.15.47:0/3124110502 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ebc01a8b0 con 0x7f1ec80724d0 2024-01-18T06:47:36.238 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.236+0000 7f1ecee8b700 1 -- 172.21.15.47:0/3124110502 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ec810d3d0 con 0x7f1ec80724d0 2024-01-18T06:47:36.238 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.236+0000 7f1ecee8b700 1 -- 172.21.15.47:0/3124110502 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ec810d7e0 con 0x7f1ec80724d0 2024-01-18T06:47:36.238 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.236+0000 7f1ec57fa700 1 -- 172.21.15.47:0/3124110502 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f1ebc00f070 con 0x7f1ec80724d0 2024-01-18T06:47:36.238 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.236+0000 7f1ec57fa700 1 -- 172.21.15.47:0/3124110502 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f1ebc025c10 con 0x7f1ec80724d0 2024-01-18T06:47:36.238 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.237+0000 7f1ecee8b700 1 -- 172.21.15.47:0/3124110502 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ec8045ad0 con 0x7f1ec80724d0 2024-01-18T06:47:36.239 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.239+0000 7f1ec57fa700 1 -- 172.21.15.47:0/3124110502 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f1ebc016070 con 0x7f1ec80724d0 2024-01-18T06:47:36.306 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.306+0000 7f1ecee8b700 1 -- 172.21.15.47:0/3124110502 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f1ec8101e00 con 0x7f1ec80724d0 2024-01-18T06:47:36.307 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.306+0000 7f1ec57fa700 1 -- 172.21.15.47:0/3124110502 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f1ebc01ae00 con 0x7f1ec80724d0 2024-01-18T06:47:36.308 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.307+0000 7f1ecee8b700 1 -- 172.21.15.47:0/3124110502 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ec80724d0 msgr2=0x7f1ec810cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:47:36.308 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.307+0000 7f1ecee8b700 1 --2- 172.21.15.47:0/3124110502 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ec80724d0 0x7f1ec810cce0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f1ebc011820 tx=0x7f1ebc006ee0).stop 2024-01-18T06:47:36.308 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.307+0000 7f1ecee8b700 1 -- 172.21.15.47:0/3124110502 shutdown_connections 2024-01-18T06:47:36.308 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.307+0000 7f1ecee8b700 1 --2- 172.21.15.47:0/3124110502 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ec80724d0 0x7f1ec810cce0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:47:36.308 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.307+0000 7f1ecee8b700 1 -- 172.21.15.47:0/3124110502 >> 172.21.15.47:0/3124110502 conn(0x7f1ec80f74c0 msgr2=0x7f1ec80f8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:47:36.308 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.307+0000 7f1ecee8b700 1 -- 172.21.15.47:0/3124110502 shutdown_connections 2024-01-18T06:47:36.309 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:36.307+0000 7f1ecee8b700 1 -- 172.21.15.47:0/3124110502 wait complete. 2024-01-18T06:47:36.769 INFO:teuthology.orchestra.run.smithi047.stderr:Restarting the monitor... 2024-01-18T06:47:36.769 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl restart ceph-653e4012-b5cd-11ee-95ae-87774f69a715@mon.smithi047 2024-01-18T06:47:36.963 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:36 smithi047 systemd[1]: Stopping Ceph mon.smithi047 for 653e4012-b5cd-11ee-95ae-87774f69a715... 2024-01-18T06:47:36.963 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:36 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38390]: debug 2024-01-18T06:47:36.850+0000 7fe03a4d2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi047 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-01-18T06:47:36.963 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:36 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38390]: debug 2024-01-18T06:47:36.850+0000 7fe03a4d2700 -1 mon.smithi047@0(leader) e1 *** Got Signal Terminated *** 2024-01-18T06:47:37.558 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 podman[38711]: ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047 2024-01-18T06:47:37.814 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 systemd[1]: ceph-653e4012-b5cd-11ee-95ae-87774f69a715@mon.smithi047.service: Succeeded. 2024-01-18T06:47:37.814 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 systemd[1]: Stopped Ceph mon.smithi047 for 653e4012-b5cd-11ee-95ae-87774f69a715. 2024-01-18T06:47:37.814 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 systemd[1]: Starting Ceph mon.smithi047 for 653e4012-b5cd-11ee-95ae-87774f69a715... 2024-01-18T06:47:37.894 INFO:teuthology.orchestra.run.smithi047.stderr:Setting mon public_network to 172.21.0.0/20 2024-01-18T06:47:37.894 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/653e4012-b5cd-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpck5s1jz7:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcg0aieo2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2024-01-18T06:47:38.214 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.887+0000 7f4ec49a7700 0 set uid:gid to 167:167 (ceph:ceph) 2024-01-18T06:47:38.214 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.887+0000 7f4ec49a7700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2024-01-18T06:47:38.214 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.887+0000 7f4ec49a7700 0 pidfile_write: ignore empty --pid-file 2024-01-18T06:47:38.214 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 bash[38783]: dd728c664357a0431766df2f7c7748391c79d6e35c807011d044e6c6b6f5dbe0 2024-01-18T06:47:38.214 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.891+0000 7f4ec49a7700 0 load: jerasure load: lrc load: isa 2024-01-18T06:47:38.214 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: RocksDB version: 6.8.1 2024-01-18T06:47:38.214 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: 2024-01-18T06:47:38.214 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-18T06:47:38.215 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Compile date May 13 2021 2024-01-18T06:47:38.215 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: DB SUMMARY 2024-01-18T06:47:38.215 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: 2024-01-18T06:47:38.215 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: CURRENT file: CURRENT 2024-01-18T06:47:38.215 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: 2024-01-18T06:47:38.215 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: IDENTITY file: IDENTITY 2024-01-18T06:47:38.215 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: 2024-01-18T06:47:38.215 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2024-01-18T06:47:38.215 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: 2024-01-18T06:47:38.215 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi047/store.db dir, Total Num: 1, files: 000004.sst 2024-01-18T06:47:38.215 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: 2024-01-18T06:47:38.215 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi047/store.db: 000006.log size: 92918 ; 2024-01-18T06:47:38.215 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: 2024-01-18T06:47:38.215 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.error_if_exists: 0 2024-01-18T06:47:38.215 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.create_if_missing: 0 2024-01-18T06:47:38.215 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.paranoid_checks: 1 2024-01-18T06:47:38.215 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.env: 0x55ac6b3441c0 2024-01-18T06:47:38.216 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.fs: Posix File System 2024-01-18T06:47:38.216 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.info_log: 0x55ac6c3c7960 2024-01-18T06:47:38.216 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.max_file_opening_threads: 16 2024-01-18T06:47:38.216 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.statistics: (nil) 2024-01-18T06:47:38.216 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.use_fsync: 0 2024-01-18T06:47:38.216 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.max_log_file_size: 0 2024-01-18T06:47:38.216 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-18T06:47:38.216 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-01-18T06:47:38.216 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.keep_log_file_num: 1000 2024-01-18T06:47:38.216 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.recycle_log_file_num: 0 2024-01-18T06:47:38.216 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.allow_fallocate: 1 2024-01-18T06:47:38.216 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.allow_mmap_reads: 0 2024-01-18T06:47:38.216 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.allow_mmap_writes: 0 2024-01-18T06:47:38.216 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.use_direct_reads: 0 2024-01-18T06:47:38.216 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-18T06:47:38.217 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.create_missing_column_families: 0 2024-01-18T06:47:38.217 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.db_log_dir: 2024-01-18T06:47:38.217 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi047/store.db 2024-01-18T06:47:38.217 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-01-18T06:47:38.217 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.max_subcompactions: 1 2024-01-18T06:47:38.217 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.max_background_flushes: -1 2024-01-18T06:47:38.217 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-01-18T06:47:38.217 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-01-18T06:47:38.217 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-18T06:47:38.217 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-18T06:47:38.217 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-01-18T06:47:38.217 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.advise_random_on_open: 1 2024-01-18T06:47:38.217 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.db_write_buffer_size: 0 2024-01-18T06:47:38.217 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.write_buffer_manager: 0x55ac6d03c030 2024-01-18T06:47:38.217 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-18T06:47:38.217 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-18T06:47:38.218 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-18T06:47:38.218 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-01-18T06:47:38.218 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.rate_limiter: (nil) 2024-01-18T06:47:38.218 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-18T06:47:38.218 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.wal_recovery_mode: 2 2024-01-18T06:47:38.218 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.enable_thread_tracking: 0 2024-01-18T06:47:38.218 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.enable_pipelined_write: 0 2024-01-18T06:47:38.218 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.unordered_write: 0 2024-01-18T06:47:38.218 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-18T06:47:38.218 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-18T06:47:38.218 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-18T06:47:38.218 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-18T06:47:38.218 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.row_cache: None 2024-01-18T06:47:38.218 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.wal_filter: None 2024-01-18T06:47:38.218 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-18T06:47:38.218 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.allow_ingest_behind: 0 2024-01-18T06:47:38.219 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.preserve_deletes: 0 2024-01-18T06:47:38.219 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.two_write_queues: 0 2024-01-18T06:47:38.219 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.manual_wal_flush: 0 2024-01-18T06:47:38.219 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.atomic_flush: 0 2024-01-18T06:47:38.219 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-18T06:47:38.219 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-01-18T06:47:38.219 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-01-18T06:47:38.219 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.log_readahead_size: 0 2024-01-18T06:47:38.219 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-01-18T06:47:38.219 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.max_background_jobs: 2 2024-01-18T06:47:38.219 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.max_background_compactions: -1 2024-01-18T06:47:38.219 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-18T06:47:38.219 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-18T06:47:38.219 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-01-18T06:47:38.219 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.max_total_wal_size: 0 2024-01-18T06:47:38.219 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-18T06:47:38.220 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-01-18T06:47:38.220 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-01-18T06:47:38.220 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-18T06:47:38.220 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.max_open_files: -1 2024-01-18T06:47:38.220 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.bytes_per_sync: 0 2024-01-18T06:47:38.220 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-01-18T06:47:38.220 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-01-18T06:47:38.220 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.compaction_readahead_size: 0 2024-01-18T06:47:38.220 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Compression algorithms supported: 2024-01-18T06:47:38.220 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-18T06:47:38.220 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: kZSTD supported: 0 2024-01-18T06:47:38.220 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: kXpressCompression supported: 0 2024-01-18T06:47:38.220 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: kLZ4HCCompression supported: 1 2024-01-18T06:47:38.220 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: kLZ4Compression supported: 1 2024-01-18T06:47:38.221 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: kBZip2Compression supported: 0 2024-01-18T06:47:38.221 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: kZlibCompression supported: 1 2024-01-18T06:47:38.221 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: kSnappyCompression supported: 1 2024-01-18T06:47:38.221 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-01-18T06:47:38.222 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi047/store.db/MANIFEST-000005 2024-01-18T06:47:38.222 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: 2024-01-18T06:47:38.222 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-01-18T06:47:38.222 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: 2024-01-18T06:47:38.222 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-18T06:47:38.223 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.merge_operator: 2024-01-18T06:47:38.223 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.compaction_filter: None 2024-01-18T06:47:38.223 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.compaction_filter_factory: None 2024-01-18T06:47:38.223 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-01-18T06:47:38.223 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-01-18T06:47:38.223 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ac6c2de128) 2024-01-18T06:47:38.223 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cache_index_and_filter_blocks: 1 2024-01-18T06:47:38.223 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-18T06:47:38.223 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-18T06:47:38.223 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: pin_top_level_index_and_filter: 1 2024-01-18T06:47:38.223 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: index_type: 0 2024-01-18T06:47:38.223 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: data_block_index_type: 0 2024-01-18T06:47:38.223 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: index_shortening: 1 2024-01-18T06:47:38.223 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: data_block_hash_table_util_ratio: 0.750000 2024-01-18T06:47:38.224 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: hash_index_allow_collision: 1 2024-01-18T06:47:38.224 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: checksum: 1 2024-01-18T06:47:38.224 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: no_block_cache: 0 2024-01-18T06:47:38.224 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: block_cache: 0x55ac6c314f10 2024-01-18T06:47:38.224 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: block_cache_name: BinnedLRUCache 2024-01-18T06:47:38.224 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: block_cache_options: 2024-01-18T06:47:38.224 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: capacity : 536870912 2024-01-18T06:47:38.224 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: num_shard_bits : 4 2024-01-18T06:47:38.224 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: strict_capacity_limit : 0 2024-01-18T06:47:38.224 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: high_pri_pool_ratio: 0.000 2024-01-18T06:47:38.224 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: block_cache_compressed: (nil) 2024-01-18T06:47:38.224 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: persistent_cache: (nil) 2024-01-18T06:47:38.224 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: block_size: 4096 2024-01-18T06:47:38.224 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: block_size_deviation: 10 2024-01-18T06:47:38.224 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: block_restart_interval: 16 2024-01-18T06:47:38.225 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: index_block_restart_interval: 1 2024-01-18T06:47:38.225 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: metadata_block_size: 4096 2024-01-18T06:47:38.225 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: partition_filters: 0 2024-01-18T06:47:38.225 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: use_delta_encoding: 1 2024-01-18T06:47:38.225 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-18T06:47:38.225 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: whole_key_filtering: 1 2024-01-18T06:47:38.225 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: verify_compression: 0 2024-01-18T06:47:38.225 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: read_amp_bytes_per_bit: 0 2024-01-18T06:47:38.225 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: format_version: 2 2024-01-18T06:47:38.225 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: enable_index_compression: 1 2024-01-18T06:47:38.225 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: block_align: 0 2024-01-18T06:47:38.225 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: 2024-01-18T06:47:38.225 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.write_buffer_size: 33554432 2024-01-18T06:47:38.225 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.max_write_buffer_number: 2 2024-01-18T06:47:38.225 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.compression: NoCompression 2024-01-18T06:47:38.225 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.bottommost_compression: Disabled 2024-01-18T06:47:38.226 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.prefix_extractor: nullptr 2024-01-18T06:47:38.226 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-18T06:47:38.226 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.num_levels: 7 2024-01-18T06:47:38.226 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-18T06:47:38.226 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-18T06:47:38.226 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-18T06:47:38.226 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-18T06:47:38.226 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-18T06:47:38.226 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-18T06:47:38.226 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-18T06:47:38.226 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-18T06:47:38.226 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-18T06:47:38.226 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-01-18T06:47:38.226 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.compression_opts.level: 32767 2024-01-18T06:47:38.226 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.compression_opts.strategy: 0 2024-01-18T06:47:38.226 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-18T06:47:38.227 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-18T06:47:38.227 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.compression_opts.enabled: false 2024-01-18T06:47:38.227 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-18T06:47:38.227 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-18T06:47:38.227 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-18T06:47:38.227 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.target_file_size_base: 67108864 2024-01-18T06:47:38.227 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-01-18T06:47:38.227 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-18T06:47:38.227 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-18T06:47:38.227 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-18T06:47:38.227 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-18T06:47:38.227 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-18T06:47:38.227 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-18T06:47:38.227 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-18T06:47:38.227 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-18T06:47:38.227 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-18T06:47:38.227 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-18T06:47:38.228 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-18T06:47:38.228 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-18T06:47:38.228 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.arena_block_size: 4194304 2024-01-18T06:47:38.228 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-18T06:47:38.228 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-18T06:47:38.228 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-18T06:47:38.228 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.disable_auto_compactions: 0 2024-01-18T06:47:38.228 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-18T06:47:38.228 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-18T06:47:38.228 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-18T06:47:38.228 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-18T06:47:38.228 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-18T06:47:38.228 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-18T06:47:38.228 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-18T06:47:38.228 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-18T06:47:38.229 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-18T06:47:38.229 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-18T06:47:38.229 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.table_properties_collectors: 2024-01-18T06:47:38.229 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.inplace_update_support: 0 2024-01-18T06:47:38.229 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-01-18T06:47:38.229 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-18T06:47:38.229 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-18T06:47:38.229 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-01-18T06:47:38.229 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.bloom_locality: 0 2024-01-18T06:47:38.229 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.max_successive_merges: 0 2024-01-18T06:47:38.229 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-01-18T06:47:38.229 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.paranoid_file_checks: 0 2024-01-18T06:47:38.229 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.force_consistency_checks: 0 2024-01-18T06:47:38.229 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.report_bg_io_stats: 0 2024-01-18T06:47:38.229 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.ttl: 2592000 2024-01-18T06:47:38.229 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.892+0000 7f4ec49a7700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-01-18T06:47:38.230 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.893+0000 7f4ec49a7700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi047/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-18T06:47:38.230 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: 2024-01-18T06:47:38.230 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.893+0000 7f4ec49a7700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2024-01-18T06:47:38.230 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: 2024-01-18T06:47:38.230 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.893+0000 7f4ec49a7700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705560457894004, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-01-18T06:47:38.230 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.893+0000 7f4ec49a7700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2024-01-18T06:47:38.230 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 systemd[1]: Started Ceph mon.smithi047 for 653e4012-b5cd-11ee-95ae-87774f69a715. 2024-01-18T06:47:38.230 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.893+0000 7f4ec49a7700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-01-18T06:47:38.230 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.894+0000 7f4ec49a7700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705560457895067, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 88698, "table_properties": {"data_size": 86813, "index_size": 362, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12384, "raw_average_key_size": 49, "raw_value_size": 79954, "raw_average_value_size": 321, "num_data_blocks": 11, "num_entries": 249, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1705560457, "oldest_key_time": 3, "file_creation_time": 0}} 2024-01-18T06:47:38.230 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.894+0000 7f4ec49a7700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2024-01-18T06:47:38.230 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: 2024-01-18T06:47:38.230 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.894+0000 7f4ec49a7700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705560457895479, "job": 1, "event": "recovery_finished"} 2024-01-18T06:47:38.230 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.895+0000 7f4ec49a7700 4 rocksdb: DB pointer 0x55ac6c3d9800 2024-01-18T06:47:38.230 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.895+0000 7f4eab7bf700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-01-18T06:47:38.230 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.895+0000 7f4eab7bf700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-01-18T06:47:38.230 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: ** DB Stats ** 2024-01-18T06:47:38.231 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-18T06:47:38.231 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-18T06:47:38.231 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-18T06:47:38.231 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-18T06:47:38.231 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-18T06:47:38.231 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-18T06:47:38.231 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-18T06:47:38.231 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: 2024-01-18T06:47:38.231 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: ** Compaction Stats [default] ** 2024-01-18T06:47:38.231 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-18T06:47:38.231 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-18T06:47:38.231 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: L0 2/0 88.31 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 114.2 0.00 0.00 1 0.001 0 0 2024-01-18T06:47:38.231 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: Sum 2/0 88.31 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 114.2 0.00 0.00 1 0.001 0 0 2024-01-18T06:47:38.231 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 114.2 0.00 0.00 1 0.001 0 0 2024-01-18T06:47:38.231 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: 2024-01-18T06:47:38.232 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: ** Compaction Stats [default] ** 2024-01-18T06:47:38.232 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-18T06:47:38.232 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-18T06:47:38.232 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 114.2 0.00 0.00 1 0.001 0 0 2024-01-18T06:47:38.232 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-18T06:47:38.232 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-18T06:47:38.232 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-18T06:47:38.232 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: AddFile(Total Files): cumulative 0, interval 0 2024-01-18T06:47:38.232 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-18T06:47:38.232 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: AddFile(Keys): cumulative 0, interval 0 2024-01-18T06:47:38.232 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: Cumulative compaction: 0.00 GB write, 27.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-18T06:47:38.232 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: Interval compaction: 0.00 GB write, 27.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-18T06:47:38.232 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-18T06:47:38.232 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: 2024-01-18T06:47:38.232 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: ** File Read Latency Histogram By Level [default] ** 2024-01-18T06:47:38.233 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: 2024-01-18T06:47:38.233 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: ** Compaction Stats [default] ** 2024-01-18T06:47:38.233 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-18T06:47:38.233 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-18T06:47:38.233 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: L0 2/0 88.31 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 114.2 0.00 0.00 1 0.001 0 0 2024-01-18T06:47:38.233 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: Sum 2/0 88.31 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 114.2 0.00 0.00 1 0.001 0 0 2024-01-18T06:47:38.233 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-18T06:47:38.233 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: 2024-01-18T06:47:38.233 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: ** Compaction Stats [default] ** 2024-01-18T06:47:38.233 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-18T06:47:38.233 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-18T06:47:38.233 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 114.2 0.00 0.00 1 0.001 0 0 2024-01-18T06:47:38.233 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-18T06:47:38.233 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-18T06:47:38.234 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-18T06:47:38.234 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: AddFile(Total Files): cumulative 0, interval 0 2024-01-18T06:47:38.234 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-18T06:47:38.234 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: AddFile(Keys): cumulative 0, interval 0 2024-01-18T06:47:38.234 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: Cumulative compaction: 0.00 GB write, 26.86 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-18T06:47:38.234 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-18T06:47:38.234 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-18T06:47:38.235 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: 2024-01-18T06:47:38.235 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: ** File Read Latency Histogram By Level [default] ** 2024-01-18T06:47:38.235 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: 2024-01-18T06:47:38.235 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.896+0000 7f4ec49a7700 0 starting mon.smithi047 rank 0 at public addrs [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] at bind addrs [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi047 fsid 653e4012-b5cd-11ee-95ae-87774f69a715 2024-01-18T06:47:38.235 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.897+0000 7f4ec49a7700 1 mon.smithi047@-1(???) e1 preinit fsid 653e4012-b5cd-11ee-95ae-87774f69a715 2024-01-18T06:47:38.235 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.897+0000 7f4ec49a7700 0 mon.smithi047@-1(???).mds e1 new map 2024-01-18T06:47:38.235 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.897+0000 7f4ec49a7700 0 mon.smithi047@-1(???).mds e1 print_map 2024-01-18T06:47:38.235 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: e1 2024-01-18T06:47:38.235 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: enable_multiple, ever_enabled_multiple: 1,1 2024-01-18T06:47:38.235 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-18T06:47:38.235 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: legacy client fscid: -1 2024-01-18T06:47:38.235 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: 2024-01-18T06:47:38.235 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: No filesystems configured 2024-01-18T06:47:38.235 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: 2024-01-18T06:47:38.235 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.897+0000 7f4ec49a7700 0 mon.smithi047@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-01-18T06:47:38.235 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.897+0000 7f4ec49a7700 0 mon.smithi047@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-18T06:47:38.236 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.897+0000 7f4ec49a7700 0 mon.smithi047@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-18T06:47:38.236 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.897+0000 7f4ec49a7700 0 mon.smithi047@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-18T06:47:38.236 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:47:35.510067+0000 mon.smithi047 (mon.0) 8 : audit [INF] from='client.? 172.21.15.47:0/4111803396' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-18T06:47:38.236 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:47:35.513128+0000 mon.smithi047 (mon.0) 9 : audit [INF] from='client.? 172.21.15.47:0/4111803396' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-01-18T06:47:38.236 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.897+0000 7f4ec49a7700 1 mon.smithi047@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-01-18T06:47:38.236 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.898+0000 7f4ec49a7700 20 expand_channel_meta expand map: {default=false} 2024-01-18T06:47:38.236 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.898+0000 7f4ec49a7700 20 expand_channel_meta from 'false' to 'false' 2024-01-18T06:47:38.236 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.898+0000 7f4ec49a7700 20 expand_channel_meta expanded map: {default=false} 2024-01-18T06:47:38.236 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.898+0000 7f4ec49a7700 20 expand_channel_meta expand map: {default=info} 2024-01-18T06:47:38.236 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.898+0000 7f4ec49a7700 20 expand_channel_meta from 'info' to 'info' 2024-01-18T06:47:38.236 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.898+0000 7f4ec49a7700 20 expand_channel_meta expanded map: {default=info} 2024-01-18T06:47:38.236 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.898+0000 7f4ec49a7700 20 expand_channel_meta expand map: {default=daemon} 2024-01-18T06:47:38.236 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.898+0000 7f4ec49a7700 20 expand_channel_meta from 'daemon' to 'daemon' 2024-01-18T06:47:38.236 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.898+0000 7f4ec49a7700 20 expand_channel_meta expanded map: {default=daemon} 2024-01-18T06:47:38.236 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.898+0000 7f4ec49a7700 20 expand_channel_meta expand map: {default=debug} 2024-01-18T06:47:38.237 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.898+0000 7f4ec49a7700 20 expand_channel_meta from 'debug' to 'debug' 2024-01-18T06:47:38.237 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:47:37.898+0000 7f4ec49a7700 20 expand_channel_meta expanded map: {default=debug} 2024-01-18T06:47:38.237 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:47:37.914187+0000 mon.smithi047 (mon.0) 1 : cluster [INF] mon.smithi047 is new leader, mons smithi047 in quorum (ranks 0) 2024-01-18T06:47:38.237 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:47:37.914293+0000 mon.smithi047 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi047=[v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0]} 2024-01-18T06:47:38.237 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:47:37.914337+0000 mon.smithi047 (mon.0) 3 : cluster [DBG] fsmap 2024-01-18T06:47:38.237 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:47:37.914370+0000 mon.smithi047 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-01-18T06:47:38.237 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:47:37.914914+0000 mon.smithi047 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-01-18T06:47:38.259 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.259+0000 7f6c83fff700 1 Processor -- start 2024-01-18T06:47:38.259 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.259+0000 7f6c83fff700 1 -- start start 2024-01-18T06:47:38.260 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.259+0000 7f6c83fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c840fd100 0x7f6c840fd520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T06:47:38.260 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.259+0000 7f6c83fff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c840fda10 con 0x7f6c840fd100 2024-01-18T06:47:38.261 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.260+0000 7f6c82ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c840fd100 0x7f6c840fd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:47:38.261 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.260+0000 7f6c82ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c840fd100 0x7f6c840fd520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:42794/0 (socket says 172.21.15.47:42794) 2024-01-18T06:47:38.261 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.260+0000 7f6c82ffd700 1 -- 172.21.15.47:0/1027894868 learned_addr learned my addr 172.21.15.47:0/1027894868 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:47:38.261 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.260+0000 7f6c82ffd700 1 -- 172.21.15.47:0/1027894868 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c840fdb50 con 0x7f6c840fd100 2024-01-18T06:47:38.261 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.260+0000 7f6c82ffd700 1 --2- 172.21.15.47:0/1027894868 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c840fd100 0x7f6c840fd520 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f6c7400b5f0 tx=0x7f6c74006ee0).ready entity=mon.0 client_cookie=501d54f3aaf481a9 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:47:38.261 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.260+0000 7f6c81ffb700 1 -- 172.21.15.47:0/1027894868 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c7400d600 con 0x7f6c840fd100 2024-01-18T06:47:38.261 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.260+0000 7f6c81ffb700 1 -- 172.21.15.47:0/1027894868 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f6c7400dbd0 con 0x7f6c840fd100 2024-01-18T06:47:38.261 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.260+0000 7f6c81ffb700 1 -- 172.21.15.47:0/1027894868 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c74011c00 con 0x7f6c840fd100 2024-01-18T06:47:38.263 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.261+0000 7f6c83fff700 1 -- 172.21.15.47:0/1027894868 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c840fd100 msgr2=0x7f6c840fd520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:47:38.264 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.261+0000 7f6c83fff700 1 --2- 172.21.15.47:0/1027894868 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c840fd100 0x7f6c840fd520 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f6c7400b5f0 tx=0x7f6c74006ee0).stop 2024-01-18T06:47:38.264 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.261+0000 7f6c83fff700 1 -- 172.21.15.47:0/1027894868 shutdown_connections 2024-01-18T06:47:38.264 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.261+0000 7f6c83fff700 1 --2- 172.21.15.47:0/1027894868 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c840fd100 0x7f6c840fd520 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:47:38.264 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.261+0000 7f6c83fff700 1 -- 172.21.15.47:0/1027894868 >> 172.21.15.47:0/1027894868 conn(0x7f6c840f59a0 msgr2=0x7f6c840f7dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:47:38.264 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.261+0000 7f6c83fff700 1 -- 172.21.15.47:0/1027894868 shutdown_connections 2024-01-18T06:47:38.264 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.261+0000 7f6c83fff700 1 -- 172.21.15.47:0/1027894868 wait complete. 2024-01-18T06:47:38.264 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.262+0000 7f6c83fff700 1 Processor -- start 2024-01-18T06:47:38.264 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.262+0000 7f6c83fff700 1 -- start start 2024-01-18T06:47:38.264 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.262+0000 7f6c83fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c840fd100 0x7f6c840fc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:47:38.264 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.262+0000 7f6c83fff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c840fae50 con 0x7f6c840fd100 2024-01-18T06:47:38.264 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.262+0000 7f6c82ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c840fd100 0x7f6c840fc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:47:38.264 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.263+0000 7f6c82ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c840fd100 0x7f6c840fc9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:42796/0 (socket says 172.21.15.47:42796) 2024-01-18T06:47:38.264 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.263+0000 7f6c82ffd700 1 -- 172.21.15.47:0/3020544211 learned_addr learned my addr 172.21.15.47:0/3020544211 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:47:38.265 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.263+0000 7f6c82ffd700 1 -- 172.21.15.47:0/3020544211 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c7400b200 con 0x7f6c840fd100 2024-01-18T06:47:38.265 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.263+0000 7f6c82ffd700 1 --2- 172.21.15.47:0/3020544211 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c840fd100 0x7f6c840fc9d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f6c74006c70 tx=0x7f6c74006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:47:38.265 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.263+0000 7f6c6b7fe700 1 -- 172.21.15.47:0/3020544211 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c74011690 con 0x7f6c840fd100 2024-01-18T06:47:38.265 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.263+0000 7f6c83fff700 1 -- 172.21.15.47:0/3020544211 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c840faff0 con 0x7f6c840fd100 2024-01-18T06:47:38.265 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.263+0000 7f6c6b7fe700 1 -- 172.21.15.47:0/3020544211 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f6c7401b880 con 0x7f6c840fd100 2024-01-18T06:47:38.265 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.263+0000 7f6c6b7fe700 1 -- 172.21.15.47:0/3020544211 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c7401a650 con 0x7f6c840fd100 2024-01-18T06:47:38.265 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.263+0000 7f6c83fff700 1 -- 172.21.15.47:0/3020544211 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c840fb400 con 0x7f6c840fd100 2024-01-18T06:47:38.265 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.263+0000 7f6c6b7fe700 1 -- 172.21.15.47:0/3020544211 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f6c7400f070 con 0x7f6c840fd100 2024-01-18T06:47:38.265 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.263+0000 7f6c6b7fe700 1 -- 172.21.15.47:0/3020544211 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f6c74025a40 con 0x7f6c840fd100 2024-01-18T06:47:38.265 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.264+0000 7f6c83fff700 1 -- 172.21.15.47:0/3020544211 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c84045ad0 con 0x7f6c840fd100 2024-01-18T06:47:38.266 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.266+0000 7f6c6b7fe700 1 -- 172.21.15.47:0/3020544211 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f6c74016070 con 0x7f6c840fd100 2024-01-18T06:47:38.332 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.333+0000 7f6c83fff700 1 -- 172.21.15.47:0/3020544211 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f6c84027a00 con 0x7f6c840fd100 2024-01-18T06:47:38.334 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.335+0000 7f6c6b7fe700 1 -- 172.21.15.47:0/3020544211 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f6c7401bd60 con 0x7f6c840fd100 2024-01-18T06:47:38.335 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.336+0000 7f6c83fff700 1 -- 172.21.15.47:0/3020544211 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c840fd100 msgr2=0x7f6c840fc9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:47:38.336 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.336+0000 7f6c83fff700 1 --2- 172.21.15.47:0/3020544211 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c840fd100 0x7f6c840fc9d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f6c74006c70 tx=0x7f6c74006ee0).stop 2024-01-18T06:47:38.336 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.336+0000 7f6c83fff700 1 -- 172.21.15.47:0/3020544211 shutdown_connections 2024-01-18T06:47:38.336 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.336+0000 7f6c83fff700 1 --2- 172.21.15.47:0/3020544211 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c840fd100 0x7f6c840fc9d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:47:38.336 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.336+0000 7f6c83fff700 1 -- 172.21.15.47:0/3020544211 >> 172.21.15.47:0/3020544211 conn(0x7f6c840f59a0 msgr2=0x7f6c840ff180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:47:38.336 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.336+0000 7f6c83fff700 1 -- 172.21.15.47:0/3020544211 shutdown_connections 2024-01-18T06:47:38.336 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:38.336+0000 7f6c83fff700 1 -- 172.21.15.47:0/3020544211 wait complete. 2024-01-18T06:47:38.694 INFO:teuthology.orchestra.run.smithi047.stderr:Wrote config to /etc/ceph/ceph.conf 2024-01-18T06:47:38.694 INFO:teuthology.orchestra.run.smithi047.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-01-18T06:47:38.694 INFO:teuthology.orchestra.run.smithi047.stderr:Creating mgr... 2024-01-18T06:47:38.695 INFO:teuthology.orchestra.run.smithi047.stderr:Verifying port 9283 ... 2024-01-18T06:47:38.696 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl daemon-reload 2024-01-18T06:47:38.801 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl stop ceph-653e4012-b5cd-11ee-95ae-87774f69a715@mgr.smithi047.fuhzzr 2024-01-18T06:47:38.834 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl reset-failed ceph-653e4012-b5cd-11ee-95ae-87774f69a715@mgr.smithi047.fuhzzr 2024-01-18T06:47:38.867 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: Failed to reset failed state of unit ceph-653e4012-b5cd-11ee-95ae-87774f69a715@mgr.smithi047.fuhzzr.service: Unit ceph-653e4012-b5cd-11ee-95ae-87774f69a715@mgr.smithi047.fuhzzr.service not loaded. 2024-01-18T06:47:38.868 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl enable ceph-653e4012-b5cd-11ee-95ae-87774f69a715@mgr.smithi047.fuhzzr 2024-01-18T06:47:38.868 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: Created symlink /etc/systemd/system/ceph-653e4012-b5cd-11ee-95ae-87774f69a715.target.wants/ceph-653e4012-b5cd-11ee-95ae-87774f69a715@mgr.smithi047.fuhzzr.service → /etc/systemd/system/ceph-653e4012-b5cd-11ee-95ae-87774f69a715@.service. 2024-01-18T06:47:38.939 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl start ceph-653e4012-b5cd-11ee-95ae-87774f69a715@mgr.smithi047.fuhzzr 2024-01-18T06:47:39.479 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-18T06:47:39.488 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: disabled 2024-01-18T06:47:39.488 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl is-active firewalld.service 2024-01-18T06:47:39.496 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: inactive 2024-01-18T06:47:39.497 INFO:teuthology.orchestra.run.smithi047.stderr:firewalld.service is not enabled 2024-01-18T06:47:39.497 INFO:teuthology.orchestra.run.smithi047.stderr:Not possible to enable service . firewalld.service is not available 2024-01-18T06:47:39.497 INFO:teuthology.orchestra.run.smithi047.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-01-18T06:47:39.497 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-18T06:47:39.504 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: disabled 2024-01-18T06:47:39.505 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl is-active firewalld.service 2024-01-18T06:47:39.512 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: inactive 2024-01-18T06:47:39.514 INFO:teuthology.orchestra.run.smithi047.stderr:firewalld.service is not enabled 2024-01-18T06:47:39.514 INFO:teuthology.orchestra.run.smithi047.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-01-18T06:47:39.514 INFO:teuthology.orchestra.run.smithi047.stderr:Waiting for mgr to start... 2024-01-18T06:47:39.514 INFO:teuthology.orchestra.run.smithi047.stderr:Waiting for mgr... 2024-01-18T06:47:39.514 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/653e4012-b5cd-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpck5s1jz7:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcg0aieo2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-01-18T06:47:39.617 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:39 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:47:38.335899+0000 mon.smithi047 (mon.0) 6 : audit [INF] from='client.? 172.21.15.47:0/3020544211' entity='client.admin' 2024-01-18T06:47:39.880 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.878+0000 7f27f8d60700 1 Processor -- start 2024-01-18T06:47:39.880 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.878+0000 7f27f8d60700 1 -- start start 2024-01-18T06:47:39.880 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.879+0000 7f27f8d60700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27f40703f0 0x7f27f40fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T06:47:39.880 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.879+0000 7f27f8d60700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f40fee10 con 0x7f27f40703f0 2024-01-18T06:47:39.880 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.879+0000 7f27f259c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27f40703f0 0x7f27f40fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:47:39.880 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.879+0000 7f27f259c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27f40703f0 0x7f27f40fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:42828/0 (socket says 172.21.15.47:42828) 2024-01-18T06:47:39.880 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.879+0000 7f27f259c700 1 -- 172.21.15.47:0/3603022264 learned_addr learned my addr 172.21.15.47:0/3603022264 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:47:39.882 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.879+0000 7f27f259c700 1 -- 172.21.15.47:0/3603022264 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f40fef50 con 0x7f27f40703f0 2024-01-18T06:47:39.882 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.880+0000 7f27f259c700 1 --2- 172.21.15.47:0/3603022264 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27f40703f0 0x7f27f40fe920 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f27e400b5f0 tx=0x7f27e4006ee0).ready entity=mon.0 client_cookie=7b12b2262492f91e server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:47:39.882 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.880+0000 7f27f159a700 1 -- 172.21.15.47:0/3603022264 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27e400d600 con 0x7f27f40703f0 2024-01-18T06:47:39.883 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.880+0000 7f27f159a700 1 -- 172.21.15.47:0/3603022264 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f27e400dbd0 con 0x7f27f40703f0 2024-01-18T06:47:39.883 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.880+0000 7f27f159a700 1 -- 172.21.15.47:0/3603022264 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27e4011c00 con 0x7f27f40703f0 2024-01-18T06:47:39.883 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.880+0000 7f27f8d60700 1 -- 172.21.15.47:0/3603022264 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27f40703f0 msgr2=0x7f27f40fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:47:39.883 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.880+0000 7f27f8d60700 1 --2- 172.21.15.47:0/3603022264 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27f40703f0 0x7f27f40fe920 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f27e400b5f0 tx=0x7f27e4006ee0).stop 2024-01-18T06:47:39.883 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.881+0000 7f27f8d60700 1 -- 172.21.15.47:0/3603022264 shutdown_connections 2024-01-18T06:47:39.883 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.881+0000 7f27f8d60700 1 --2- 172.21.15.47:0/3603022264 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27f40703f0 0x7f27f40fe920 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:47:39.883 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.881+0000 7f27f8d60700 1 -- 172.21.15.47:0/3603022264 >> 172.21.15.47:0/3603022264 conn(0x7f27f40f63c0 msgr2=0x7f27f40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:47:39.883 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.881+0000 7f27f8d60700 1 -- 172.21.15.47:0/3603022264 shutdown_connections 2024-01-18T06:47:39.883 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.881+0000 7f27f8d60700 1 -- 172.21.15.47:0/3603022264 wait complete. 2024-01-18T06:47:39.883 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.881+0000 7f27f8d60700 1 Processor -- start 2024-01-18T06:47:39.883 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.881+0000 7f27f8d60700 1 -- start start 2024-01-18T06:47:39.883 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.882+0000 7f27f8d60700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27f40703f0 0x7f27f410a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:47:39.884 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.882+0000 7f27f8d60700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f410aeb0 con 0x7f27f40703f0 2024-01-18T06:47:39.884 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.882+0000 7f27f259c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27f40703f0 0x7f27f410a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:47:39.884 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.882+0000 7f27f259c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27f40703f0 0x7f27f410a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:42832/0 (socket says 172.21.15.47:42832) 2024-01-18T06:47:39.884 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.882+0000 7f27f259c700 1 -- 172.21.15.47:0/3528695266 learned_addr learned my addr 172.21.15.47:0/3528695266 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:47:39.884 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.882+0000 7f27f259c700 1 -- 172.21.15.47:0/3528695266 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e400b200 con 0x7f27f40703f0 2024-01-18T06:47:39.884 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.882+0000 7f27f259c700 1 --2- 172.21.15.47:0/3528695266 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27f40703f0 0x7f27f410a9c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f27e400b190 tx=0x7f27e4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:47:39.884 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.882+0000 7f27e2ffd700 1 -- 172.21.15.47:0/3528695266 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27e401b410 con 0x7f27f40703f0 2024-01-18T06:47:39.884 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.883+0000 7f27e2ffd700 1 -- 172.21.15.47:0/3528695266 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f27e401b9e0 con 0x7f27f40703f0 2024-01-18T06:47:39.884 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.883+0000 7f27f8d60700 1 -- 172.21.15.47:0/3528695266 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27f410b0b0 con 0x7f27f40703f0 2024-01-18T06:47:39.884 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.883+0000 7f27f8d60700 1 -- 172.21.15.47:0/3528695266 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27f410b4c0 con 0x7f27f40703f0 2024-01-18T06:47:39.885 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.883+0000 7f27e2ffd700 1 -- 172.21.15.47:0/3528695266 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27e401a880 con 0x7f27f40703f0 2024-01-18T06:47:39.885 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.884+0000 7f27e2ffd700 1 -- 172.21.15.47:0/3528695266 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f27e400f070 con 0x7f27f40703f0 2024-01-18T06:47:39.885 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.884+0000 7f27e2ffd700 1 -- 172.21.15.47:0/3528695266 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f27e40258a0 con 0x7f27f40703f0 2024-01-18T06:47:39.885 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.884+0000 7f27f8d60700 1 -- 172.21.15.47:0/3528695266 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27d0004fa0 con 0x7f27f40703f0 2024-01-18T06:47:39.887 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.888+0000 7f27e2ffd700 1 -- 172.21.15.47:0/3528695266 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f27e4016070 con 0x7f27f40703f0 2024-01-18T06:47:39.963 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.957+0000 7f27f8d60700 1 -- 172.21.15.47:0/3528695266 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f27d0005670 con 0x7f27f40703f0 2024-01-18T06:47:39.963 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.958+0000 7f27e2ffd700 1 -- 172.21.15.47:0/3528695266 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f27e401aaf0 con 0x7f27f40703f0 2024-01-18T06:47:39.963 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.963 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: { 2024-01-18T06:47:39.963 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "fsid": "653e4012-b5cd-11ee-95ae-87774f69a715", 2024-01-18T06:47:39.963 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "health": { 2024-01-18T06:47:39.963 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-18T06:47:39.964 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "checks": {}, 2024-01-18T06:47:39.964 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "mutes": [] 2024-01-18T06:47:39.964 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-01-18T06:47:39.964 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-18T06:47:39.964 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "quorum": [ 2024-01-18T06:47:39.964 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 0 2024-01-18T06:47:39.964 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: ], 2024-01-18T06:47:39.964 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-18T06:47:39.964 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "smithi047" 2024-01-18T06:47:39.964 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: ], 2024-01-18T06:47:39.964 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-01-18T06:47:39.964 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "monmap": { 2024-01-18T06:47:39.964 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-18T06:47:39.964 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-18T06:47:39.965 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-18T06:47:39.965 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-01-18T06:47:39.965 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "osdmap": { 2024-01-18T06:47:39.965 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-18T06:47:39.965 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-18T06:47:39.965 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-18T06:47:39.965 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-18T06:47:39.965 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-18T06:47:39.965 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-18T06:47:39.965 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-18T06:47:39.965 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-01-18T06:47:39.965 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "pgmap": { 2024-01-18T06:47:39.965 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-18T06:47:39.965 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-18T06:47:39.965 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-18T06:47:39.966 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-18T06:47:39.966 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-18T06:47:39.966 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-18T06:47:39.966 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-18T06:47:39.966 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-18T06:47:39.966 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-01-18T06:47:39.966 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "fsmap": { 2024-01-18T06:47:39.966 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-18T06:47:39.966 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-18T06:47:39.966 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-18T06:47:39.966 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-01-18T06:47:39.966 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-18T06:47:39.966 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "available": false, 2024-01-18T06:47:39.966 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-18T06:47:39.967 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "modules": [ 2024-01-18T06:47:39.967 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "iostat", 2024-01-18T06:47:39.967 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "restful" 2024-01-18T06:47:39.967 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: ], 2024-01-18T06:47:39.967 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "services": {} 2024-01-18T06:47:39.967 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-01-18T06:47:39.967 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "servicemap": { 2024-01-18T06:47:39.967 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-18T06:47:39.967 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "modified": "2024-01-18T06:47:33.869674+0000", 2024-01-18T06:47:39.967 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "services": {} 2024-01-18T06:47:39.967 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-01-18T06:47:39.967 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-18T06:47:39.967 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: } 2024-01-18T06:47:39.967 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.958+0000 7f27f8d60700 1 -- 172.21.15.47:0/3528695266 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27f40703f0 msgr2=0x7f27f410a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:47:39.967 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.958+0000 7f27f8d60700 1 --2- 172.21.15.47:0/3528695266 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27f40703f0 0x7f27f410a9c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f27e400b190 tx=0x7f27e4006ee0).stop 2024-01-18T06:47:39.968 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.959+0000 7f27f8d60700 1 -- 172.21.15.47:0/3528695266 shutdown_connections 2024-01-18T06:47:39.968 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.959+0000 7f27f8d60700 1 --2- 172.21.15.47:0/3528695266 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f27f40703f0 0x7f27f410a9c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:47:39.968 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.959+0000 7f27f8d60700 1 -- 172.21.15.47:0/3528695266 >> 172.21.15.47:0/3528695266 conn(0x7f27f40f63c0 msgr2=0x7f27f40f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:47:39.968 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.959+0000 7f27f8d60700 1 -- 172.21.15.47:0/3528695266 shutdown_connections 2024-01-18T06:47:39.968 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:39.959+0000 7f27f8d60700 1 -- 172.21.15.47:0/3528695266 wait complete. 2024-01-18T06:47:40.359 INFO:teuthology.orchestra.run.smithi047.stderr:mgr not available, waiting (1/15)... 2024-01-18T06:47:40.713 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:40 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:47:39.958366+0000 mon.smithi047 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.47:0/3528695266' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-18T06:47:42.361 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/653e4012-b5cd-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpck5s1jz7:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcg0aieo2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-01-18T06:47:42.792 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.792+0000 7feb95233700 1 Processor -- start 2024-01-18T06:47:42.793 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.792+0000 7feb95233700 1 -- start start 2024-01-18T06:47:42.793 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.793+0000 7feb95233700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb90070430 0x7feb900fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T06:47:42.793 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.793+0000 7feb95233700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb900fedd0 con 0x7feb90070430 2024-01-18T06:47:42.797 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.793+0000 7feb8ed9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb90070430 0x7feb900fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:47:42.797 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.793+0000 7feb8ed9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb90070430 0x7feb900fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52320/0 (socket says 172.21.15.47:52320) 2024-01-18T06:47:42.797 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.793+0000 7feb8ed9d700 1 -- 172.21.15.47:0/582650657 learned_addr learned my addr 172.21.15.47:0/582650657 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:47:42.797 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.793+0000 7feb8ed9d700 1 -- 172.21.15.47:0/582650657 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb900fef10 con 0x7feb90070430 2024-01-18T06:47:42.797 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.793+0000 7feb8ed9d700 1 --2- 172.21.15.47:0/582650657 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb90070430 0x7feb900fe8e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7feb8000b5f0 tx=0x7feb80006ee0).ready entity=mon.0 client_cookie=3130f7aac6e4ce30 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:47:42.797 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.793+0000 7feb8dd9b700 1 -- 172.21.15.47:0/582650657 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb8000d600 con 0x7feb90070430 2024-01-18T06:47:42.797 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.793+0000 7feb8dd9b700 1 -- 172.21.15.47:0/582650657 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7feb8000dbd0 con 0x7feb90070430 2024-01-18T06:47:42.797 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.793+0000 7feb8dd9b700 1 -- 172.21.15.47:0/582650657 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb80011c00 con 0x7feb90070430 2024-01-18T06:47:42.797 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.794+0000 7feb95233700 1 -- 172.21.15.47:0/582650657 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb90070430 msgr2=0x7feb900fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:47:42.797 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.794+0000 7feb95233700 1 --2- 172.21.15.47:0/582650657 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb90070430 0x7feb900fe8e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7feb8000b5f0 tx=0x7feb80006ee0).stop 2024-01-18T06:47:42.797 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.794+0000 7feb95233700 1 -- 172.21.15.47:0/582650657 shutdown_connections 2024-01-18T06:47:42.798 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.794+0000 7feb95233700 1 --2- 172.21.15.47:0/582650657 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb90070430 0x7feb900fe8e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:47:42.798 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.794+0000 7feb95233700 1 -- 172.21.15.47:0/582650657 >> 172.21.15.47:0/582650657 conn(0x7feb900f63c0 msgr2=0x7feb900f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:47:42.798 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.794+0000 7feb95233700 1 -- 172.21.15.47:0/582650657 shutdown_connections 2024-01-18T06:47:42.798 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.794+0000 7feb95233700 1 -- 172.21.15.47:0/582650657 wait complete. 2024-01-18T06:47:42.798 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.795+0000 7feb95233700 1 Processor -- start 2024-01-18T06:47:42.798 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.795+0000 7feb95233700 1 -- start start 2024-01-18T06:47:42.798 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.795+0000 7feb95233700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb90070430 0x7feb9010aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:47:42.798 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.795+0000 7feb95233700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb9010aef0 con 0x7feb90070430 2024-01-18T06:47:42.798 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.795+0000 7feb8ed9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb90070430 0x7feb9010aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:47:42.798 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.795+0000 7feb8ed9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb90070430 0x7feb9010aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52336/0 (socket says 172.21.15.47:52336) 2024-01-18T06:47:42.798 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.795+0000 7feb8ed9d700 1 -- 172.21.15.47:0/1784898820 learned_addr learned my addr 172.21.15.47:0/1784898820 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:47:42.798 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.796+0000 7feb8ed9d700 1 -- 172.21.15.47:0/1784898820 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb8000b200 con 0x7feb90070430 2024-01-18T06:47:42.798 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.796+0000 7feb8ed9d700 1 --2- 172.21.15.47:0/1784898820 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb90070430 0x7feb9010aa00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7feb8000b190 tx=0x7feb80006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:47:42.798 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.796+0000 7feb777fe700 1 -- 172.21.15.47:0/1784898820 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb8001b410 con 0x7feb90070430 2024-01-18T06:47:42.799 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.796+0000 7feb777fe700 1 -- 172.21.15.47:0/1784898820 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7feb8001b9e0 con 0x7feb90070430 2024-01-18T06:47:42.799 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.796+0000 7feb95233700 1 -- 172.21.15.47:0/1784898820 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb9010b0f0 con 0x7feb90070430 2024-01-18T06:47:42.799 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.796+0000 7feb777fe700 1 -- 172.21.15.47:0/1784898820 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb8001a880 con 0x7feb90070430 2024-01-18T06:47:42.799 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.796+0000 7feb95233700 1 -- 172.21.15.47:0/1784898820 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb9010b500 con 0x7feb90070430 2024-01-18T06:47:42.799 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.796+0000 7feb777fe700 1 -- 172.21.15.47:0/1784898820 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7feb8000f070 con 0x7feb90070430 2024-01-18T06:47:42.799 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.796+0000 7feb777fe700 1 -- 172.21.15.47:0/1784898820 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7feb80025b70 con 0x7feb90070430 2024-01-18T06:47:42.799 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.797+0000 7feb95233700 1 -- 172.21.15.47:0/1784898820 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb90045ad0 con 0x7feb90070430 2024-01-18T06:47:42.799 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.799+0000 7feb777fe700 1 -- 172.21.15.47:0/1784898820 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7feb80016070 con 0x7feb90070430 2024-01-18T06:47:42.871 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.870+0000 7feb95233700 1 -- 172.21.15.47:0/1784898820 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7feb90104890 con 0x7feb90070430 2024-01-18T06:47:42.871 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.870+0000 7feb777fe700 1 -- 172.21.15.47:0/1784898820 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7feb80025db0 con 0x7feb90070430 2024-01-18T06:47:42.871 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.872 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: { 2024-01-18T06:47:42.872 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "fsid": "653e4012-b5cd-11ee-95ae-87774f69a715", 2024-01-18T06:47:42.872 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "health": { 2024-01-18T06:47:42.872 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-18T06:47:42.872 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "checks": {}, 2024-01-18T06:47:42.872 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "mutes": [] 2024-01-18T06:47:42.872 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-01-18T06:47:42.872 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-18T06:47:42.873 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "quorum": [ 2024-01-18T06:47:42.873 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 0 2024-01-18T06:47:42.874 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: ], 2024-01-18T06:47:42.874 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-18T06:47:42.874 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "smithi047" 2024-01-18T06:47:42.874 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: ], 2024-01-18T06:47:42.874 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "quorum_age": 4, 2024-01-18T06:47:42.874 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "monmap": { 2024-01-18T06:47:42.874 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-18T06:47:42.874 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-18T06:47:42.879 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-18T06:47:42.879 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-01-18T06:47:42.879 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "osdmap": { 2024-01-18T06:47:42.879 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-18T06:47:42.879 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-18T06:47:42.879 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-18T06:47:42.879 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-18T06:47:42.879 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-18T06:47:42.879 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-18T06:47:42.879 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-18T06:47:42.879 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-01-18T06:47:42.880 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "pgmap": { 2024-01-18T06:47:42.880 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-18T06:47:42.880 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-18T06:47:42.880 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-18T06:47:42.880 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-18T06:47:42.880 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-18T06:47:42.880 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-18T06:47:42.880 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-18T06:47:42.880 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-18T06:47:42.880 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-01-18T06:47:42.880 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "fsmap": { 2024-01-18T06:47:42.880 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-18T06:47:42.880 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-18T06:47:42.880 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-18T06:47:42.881 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-01-18T06:47:42.881 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-18T06:47:42.881 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "available": false, 2024-01-18T06:47:42.881 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-18T06:47:42.881 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "modules": [ 2024-01-18T06:47:42.881 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "iostat", 2024-01-18T06:47:42.881 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "restful" 2024-01-18T06:47:42.881 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: ], 2024-01-18T06:47:42.881 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "services": {} 2024-01-18T06:47:42.881 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-01-18T06:47:42.881 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "servicemap": { 2024-01-18T06:47:42.881 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-18T06:47:42.881 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "modified": "2024-01-18T06:47:33.869674+0000", 2024-01-18T06:47:42.881 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "services": {} 2024-01-18T06:47:42.881 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-01-18T06:47:42.882 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-18T06:47:42.882 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: } 2024-01-18T06:47:42.882 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.872+0000 7feb95233700 1 -- 172.21.15.47:0/1784898820 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb90070430 msgr2=0x7feb9010aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:47:42.882 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.872+0000 7feb95233700 1 --2- 172.21.15.47:0/1784898820 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb90070430 0x7feb9010aa00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7feb8000b190 tx=0x7feb80006ee0).stop 2024-01-18T06:47:42.882 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.872+0000 7feb95233700 1 -- 172.21.15.47:0/1784898820 shutdown_connections 2024-01-18T06:47:42.882 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.872+0000 7feb95233700 1 --2- 172.21.15.47:0/1784898820 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb90070430 0x7feb9010aa00 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:47:42.882 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.872+0000 7feb95233700 1 -- 172.21.15.47:0/1784898820 >> 172.21.15.47:0/1784898820 conn(0x7feb900f63c0 msgr2=0x7feb900f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:47:42.882 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.872+0000 7feb95233700 1 -- 172.21.15.47:0/1784898820 shutdown_connections 2024-01-18T06:47:42.882 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:42.872+0000 7feb95233700 1 -- 172.21.15.47:0/1784898820 wait complete. 2024-01-18T06:47:43.187 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:42 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:47:42.870810+0000 mon.smithi047 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.47:0/1784898820' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-18T06:47:43.225 INFO:teuthology.orchestra.run.smithi047.stderr:mgr not available, waiting (2/15)... 2024-01-18T06:47:44.963 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:44 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:47:44.651497+0000 mon.smithi047 (mon.0) 9 : cluster [INF] Activating manager daemon smithi047.fuhzzr 2024-01-18T06:47:44.963 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:44 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:47:44.654099+0000 mon.smithi047 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi047.fuhzzr(active, starting, since 0.00271606s) 2024-01-18T06:47:44.964 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:44 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:47:44.654942+0000 mon.smithi047 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.47:0/3043116103' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-18T06:47:44.964 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:44 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:47:44.655106+0000 mon.smithi047 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.47:0/3043116103' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-18T06:47:44.964 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:44 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:47:44.655312+0000 mon.smithi047 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.47:0/3043116103' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-18T06:47:44.964 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:44 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:47:44.655513+0000 mon.smithi047 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.47:0/3043116103' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix": "mon metadata", "id": "smithi047"}]: dispatch 2024-01-18T06:47:44.964 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:44 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:47:44.655729+0000 mon.smithi047 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.47:0/3043116103' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix": "mgr metadata", "who": "smithi047.fuhzzr", "id": "smithi047.fuhzzr"}]: dispatch 2024-01-18T06:47:44.964 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:44 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:47:44.669029+0000 mon.smithi047 (mon.0) 16 : cluster [INF] Manager daemon smithi047.fuhzzr is now available 2024-01-18T06:47:44.964 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:44 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:47:44.675487+0000 mon.smithi047 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.47:0/3043116103' entity='mgr.smithi047.fuhzzr' 2024-01-18T06:47:44.964 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:44 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:47:44.681248+0000 mon.smithi047 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.47:0/3043116103' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.fuhzzr/mirror_snapshot_schedule"}]: dispatch 2024-01-18T06:47:44.964 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:44 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:47:44.682401+0000 mon.smithi047 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.47:0/3043116103' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.fuhzzr/trash_purge_schedule"}]: dispatch 2024-01-18T06:47:44.964 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:44 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:47:44.687187+0000 mon.smithi047 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.47:0/3043116103' entity='mgr.smithi047.fuhzzr' 2024-01-18T06:47:44.964 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:44 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:47:44.689271+0000 mon.smithi047 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.47:0/3043116103' entity='mgr.smithi047.fuhzzr' 2024-01-18T06:47:45.228 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/653e4012-b5cd-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpck5s1jz7:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcg0aieo2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-01-18T06:47:45.619 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.616+0000 7ff20cea4700 1 Processor -- start 2024-01-18T06:47:45.619 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.616+0000 7ff20cea4700 1 -- start start 2024-01-18T06:47:45.619 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.616+0000 7ff20cea4700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff208070370 0x7ff2080fe9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T06:47:45.620 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.616+0000 7ff20cea4700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff208070790 con 0x7ff208070370 2024-01-18T06:47:45.620 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.616+0000 7ff20659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff208070370 0x7ff2080fe9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:47:45.620 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.617+0000 7ff20659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff208070370 0x7ff2080fe9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52392/0 (socket says 172.21.15.47:52392) 2024-01-18T06:47:45.620 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.617+0000 7ff20659c700 1 -- 172.21.15.47:0/2577665388 learned_addr learned my addr 172.21.15.47:0/2577665388 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:47:45.620 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.617+0000 7ff20659c700 1 -- 172.21.15.47:0/2577665388 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2080fee90 con 0x7ff208070370 2024-01-18T06:47:45.620 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.617+0000 7ff20659c700 1 --2- 172.21.15.47:0/2577665388 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff208070370 0x7ff2080fe9a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7ff1f800b5f0 tx=0x7ff1f8006ee0).ready entity=mon.0 client_cookie=6d71a5eb4ac33115 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:47:45.620 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.617+0000 7ff20559a700 1 -- 172.21.15.47:0/2577665388 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1f800d600 con 0x7ff208070370 2024-01-18T06:47:45.620 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.617+0000 7ff20559a700 1 -- 172.21.15.47:0/2577665388 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff1f800dbd0 con 0x7ff208070370 2024-01-18T06:47:45.620 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.617+0000 7ff20559a700 1 -- 172.21.15.47:0/2577665388 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1f8011c00 con 0x7ff208070370 2024-01-18T06:47:45.620 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.618+0000 7ff20cea4700 1 -- 172.21.15.47:0/2577665388 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff208070370 msgr2=0x7ff2080fe9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:47:45.620 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.618+0000 7ff20cea4700 1 --2- 172.21.15.47:0/2577665388 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff208070370 0x7ff2080fe9a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7ff1f800b5f0 tx=0x7ff1f8006ee0).stop 2024-01-18T06:47:45.620 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.618+0000 7ff20cea4700 1 -- 172.21.15.47:0/2577665388 shutdown_connections 2024-01-18T06:47:45.621 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.618+0000 7ff20cea4700 1 --2- 172.21.15.47:0/2577665388 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff208070370 0x7ff2080fe9a0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:47:45.621 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.618+0000 7ff20cea4700 1 -- 172.21.15.47:0/2577665388 >> 172.21.15.47:0/2577665388 conn(0x7ff2080f63c0 msgr2=0x7ff2080f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:47:45.621 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.618+0000 7ff20cea4700 1 -- 172.21.15.47:0/2577665388 shutdown_connections 2024-01-18T06:47:45.621 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.618+0000 7ff20cea4700 1 -- 172.21.15.47:0/2577665388 wait complete. 2024-01-18T06:47:45.621 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.618+0000 7ff20cea4700 1 Processor -- start 2024-01-18T06:47:45.621 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.618+0000 7ff20cea4700 1 -- start start 2024-01-18T06:47:45.621 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.619+0000 7ff20cea4700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff208070370 0x7ff2080fc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:47:45.621 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.619+0000 7ff20cea4700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2080fa8a0 con 0x7ff208070370 2024-01-18T06:47:45.621 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.619+0000 7ff20659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff208070370 0x7ff2080fc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:47:45.621 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.619+0000 7ff20659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff208070370 0x7ff2080fc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52394/0 (socket says 172.21.15.47:52394) 2024-01-18T06:47:45.621 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.619+0000 7ff20659c700 1 -- 172.21.15.47:0/3172033933 learned_addr learned my addr 172.21.15.47:0/3172033933 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:47:45.621 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.619+0000 7ff20659c700 1 -- 172.21.15.47:0/3172033933 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f800b200 con 0x7ff208070370 2024-01-18T06:47:45.622 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.619+0000 7ff20659c700 1 --2- 172.21.15.47:0/3172033933 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff208070370 0x7ff2080fc420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7ff1f800e040 tx=0x7ff1f8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:47:45.622 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.619+0000 7ff1f6ffd700 1 -- 172.21.15.47:0/3172033933 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1f800d9f0 con 0x7ff208070370 2024-01-18T06:47:45.622 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.619+0000 7ff1f6ffd700 1 -- 172.21.15.47:0/3172033933 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff1f801a9f0 con 0x7ff208070370 2024-01-18T06:47:45.622 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.619+0000 7ff20cea4700 1 -- 172.21.15.47:0/3172033933 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2080faa40 con 0x7ff208070370 2024-01-18T06:47:45.622 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.619+0000 7ff1f6ffd700 1 -- 172.21.15.47:0/3172033933 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1f801b890 con 0x7ff208070370 2024-01-18T06:47:45.622 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.619+0000 7ff20cea4700 1 -- 172.21.15.47:0/3172033933 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2080fae50 con 0x7ff208070370 2024-01-18T06:47:45.623 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.620+0000 7ff1f6ffd700 1 -- 172.21.15.47:0/3172033933 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 2) v1 ==== 35959+0+0 (secure 0 0 0) 0x7ff1f800f070 con 0x7ff208070370 2024-01-18T06:47:45.623 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.620+0000 7ff20cea4700 1 -- 172.21.15.47:0/3172033933 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff208045ad0 con 0x7ff208070370 2024-01-18T06:47:45.623 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.621+0000 7ff1f6ffd700 1 -- 172.21.15.47:0/3172033933 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7ff1f8041040 con 0x7ff208070370 2024-01-18T06:47:45.623 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.623+0000 7ff1f6ffd700 1 -- 172.21.15.47:0/3172033933 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7ff1f80224f0 con 0x7ff208070370 2024-01-18T06:47:45.654 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.653+0000 7ff1f6ffd700 1 -- 172.21.15.47:0/3172033933 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mgrmap(e 3) v1 ==== 36029+0+0 (secure 0 0 0) 0x7ff1f8022d80 con 0x7ff208070370 2024-01-18T06:47:45.654 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.654+0000 7ff1f6ffd700 1 --2- 172.21.15.47:0/3172033933 >> [v2:172.21.15.47:6800/3483218802,v1:172.21.15.47:6801/3483218802] conn(0x7ff1ec030de0 0x7ff1ec0331d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:47:45.654 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.654+0000 7ff205d9b700 1 --2- 172.21.15.47:0/3172033933 >> [v2:172.21.15.47:6800/3483218802,v1:172.21.15.47:6801/3483218802] conn(0x7ff1ec030de0 0x7ff1ec0331d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:47:45.654 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.655+0000 7ff205d9b700 1 --2- 172.21.15.47:0/3172033933 >> [v2:172.21.15.47:6800/3483218802,v1:172.21.15.47:6801/3483218802] conn(0x7ff1ec030de0 0x7ff1ec0331d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7ff1fc0096a0 tx=0x7ff1fc006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:47:45.694 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.695+0000 7ff20cea4700 1 -- 172.21.15.47:0/3172033933 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7ff208110a20 con 0x7ff208070370 2024-01-18T06:47:45.701 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.695+0000 7ff1f6ffd700 1 -- 172.21.15.47:0/3172033933 <== mon.0 v2:172.21.15.47:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1229 (secure 0 0 0) 0x7ff1f803ff50 con 0x7ff208070370 2024-01-18T06:47:45.701 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.701 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: { 2024-01-18T06:47:45.701 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "fsid": "653e4012-b5cd-11ee-95ae-87774f69a715", 2024-01-18T06:47:45.702 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "health": { 2024-01-18T06:47:45.702 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-18T06:47:45.702 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "checks": {}, 2024-01-18T06:47:45.702 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "mutes": [] 2024-01-18T06:47:45.702 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-01-18T06:47:45.702 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-18T06:47:45.702 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "quorum": [ 2024-01-18T06:47:45.702 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 0 2024-01-18T06:47:45.702 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: ], 2024-01-18T06:47:45.702 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-18T06:47:45.702 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "smithi047" 2024-01-18T06:47:45.702 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: ], 2024-01-18T06:47:45.703 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "quorum_age": 7, 2024-01-18T06:47:45.703 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "monmap": { 2024-01-18T06:47:45.703 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-18T06:47:45.703 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-18T06:47:45.703 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-18T06:47:45.703 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-01-18T06:47:45.703 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "osdmap": { 2024-01-18T06:47:45.703 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-18T06:47:45.703 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-18T06:47:45.703 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-18T06:47:45.703 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-18T06:47:45.703 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-18T06:47:45.703 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-18T06:47:45.704 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-18T06:47:45.704 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-01-18T06:47:45.704 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "pgmap": { 2024-01-18T06:47:45.704 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-18T06:47:45.704 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-18T06:47:45.704 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-18T06:47:45.704 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-18T06:47:45.704 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-18T06:47:45.704 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-18T06:47:45.704 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-18T06:47:45.704 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-18T06:47:45.704 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-01-18T06:47:45.705 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "fsmap": { 2024-01-18T06:47:45.705 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-18T06:47:45.705 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-18T06:47:45.705 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-18T06:47:45.705 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-01-18T06:47:45.705 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-18T06:47:45.705 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "available": true, 2024-01-18T06:47:45.705 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-18T06:47:45.705 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "modules": [ 2024-01-18T06:47:45.705 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "iostat", 2024-01-18T06:47:45.705 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "restful" 2024-01-18T06:47:45.705 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: ], 2024-01-18T06:47:45.706 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "services": {} 2024-01-18T06:47:45.706 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-01-18T06:47:45.706 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "servicemap": { 2024-01-18T06:47:45.706 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-18T06:47:45.706 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "modified": "2024-01-18T06:47:33.869674+0000", 2024-01-18T06:47:45.706 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "services": {} 2024-01-18T06:47:45.706 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: }, 2024-01-18T06:47:45.706 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-18T06:47:45.706 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: } 2024-01-18T06:47:45.706 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.696+0000 7ff20cea4700 1 -- 172.21.15.47:0/3172033933 >> [v2:172.21.15.47:6800/3483218802,v1:172.21.15.47:6801/3483218802] conn(0x7ff1ec030de0 msgr2=0x7ff1ec0331d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:47:45.706 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.696+0000 7ff20cea4700 1 --2- 172.21.15.47:0/3172033933 >> [v2:172.21.15.47:6800/3483218802,v1:172.21.15.47:6801/3483218802] conn(0x7ff1ec030de0 0x7ff1ec0331d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7ff1fc0096a0 tx=0x7ff1fc006b40).stop 2024-01-18T06:47:45.706 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.696+0000 7ff20cea4700 1 -- 172.21.15.47:0/3172033933 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff208070370 msgr2=0x7ff2080fc420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:47:45.706 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.696+0000 7ff20cea4700 1 --2- 172.21.15.47:0/3172033933 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff208070370 0x7ff2080fc420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7ff1f800e040 tx=0x7ff1f8006ee0).stop 2024-01-18T06:47:45.707 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.696+0000 7ff20cea4700 1 -- 172.21.15.47:0/3172033933 shutdown_connections 2024-01-18T06:47:45.707 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.696+0000 7ff20cea4700 1 --2- 172.21.15.47:0/3172033933 >> [v2:172.21.15.47:6800/3483218802,v1:172.21.15.47:6801/3483218802] conn(0x7ff1ec030de0 0x7ff1ec0331d0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:47:45.707 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.696+0000 7ff20cea4700 1 --2- 172.21.15.47:0/3172033933 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff208070370 0x7ff2080fc420 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:47:45.707 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.696+0000 7ff20cea4700 1 -- 172.21.15.47:0/3172033933 >> 172.21.15.47:0/3172033933 conn(0x7ff2080f63c0 msgr2=0x7ff2080f7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:47:45.707 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.696+0000 7ff20cea4700 1 -- 172.21.15.47:0/3172033933 shutdown_connections 2024-01-18T06:47:45.707 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:45.696+0000 7ff20cea4700 1 -- 172.21.15.47:0/3172033933 wait complete. 2024-01-18T06:47:46.067 INFO:teuthology.orchestra.run.smithi047.stderr:mgr is available 2024-01-18T06:47:46.067 INFO:teuthology.orchestra.run.smithi047.stderr:Enabling cephadm module... 2024-01-18T06:47:46.067 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/653e4012-b5cd-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpck5s1jz7:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcg0aieo2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2024-01-18T06:47:46.465 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:46.463+0000 7fd813c78700 1 Processor -- start 2024-01-18T06:47:46.465 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:46.464+0000 7fd813c78700 1 -- start start 2024-01-18T06:47:46.465 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:46.464+0000 7fd813c78700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd80c0fab90 0x7fd80c0fafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T06:47:46.465 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:46.464+0000 7fd813c78700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd80c0fb4a0 con 0x7fd80c0fab90 2024-01-18T06:47:46.465 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:46.464+0000 7fd811a14700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd80c0fab90 0x7fd80c0fafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:47:46.465 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:46.464+0000 7fd811a14700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd80c0fab90 0x7fd80c0fafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52400/0 (socket says 172.21.15.47:52400) 2024-01-18T06:47:46.465 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:46.464+0000 7fd811a14700 1 -- 172.21.15.47:0/4089959049 learned_addr learned my addr 172.21.15.47:0/4089959049 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:47:46.465 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:46.464+0000 7fd811a14700 1 -- 172.21.15.47:0/4089959049 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd80c0fb5e0 con 0x7fd80c0fab90 2024-01-18T06:47:46.466 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:46.465+0000 7fd811a14700 1 --2- 172.21.15.47:0/4089959049 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd80c0fab90 0x7fd80c0fafb0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fd80800b5f0 tx=0x7fd808006ee0).ready entity=mon.0 client_cookie=b48a8be895a73116 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:47:46.466 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:46.465+0000 7fd810a12700 1 -- 172.21.15.47:0/4089959049 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd80800d600 con 0x7fd80c0fab90 2024-01-18T06:47:46.466 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:46.465+0000 7fd810a12700 1 -- 172.21.15.47:0/4089959049 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd80800dbd0 con 0x7fd80c0fab90 2024-01-18T06:47:46.466 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:46.465+0000 7fd810a12700 1 -- 172.21.15.47:0/4089959049 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd808011c00 con 0x7fd80c0fab90 2024-01-18T06:47:46.466 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:46.465+0000 7fd813c78700 1 -- 172.21.15.47:0/4089959049 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd80c0fab90 msgr2=0x7fd80c0fafb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:47:46.466 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:46.465+0000 7fd813c78700 1 --2- 172.21.15.47:0/4089959049 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd80c0fab90 0x7fd80c0fafb0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fd80800b5f0 tx=0x7fd808006ee0).stop 2024-01-18T06:47:46.466 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:46.465+0000 7fd813c78700 1 -- 172.21.15.47:0/4089959049 shutdown_connections 2024-01-18T06:47:46.466 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:46.465+0000 7fd813c78700 1 --2- 172.21.15.47:0/4089959049 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd80c0fab90 0x7fd80c0fafb0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:47:46.466 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:46.465+0000 7fd813c78700 1 -- 172.21.15.47:0/4089959049 >> 172.21.15.47:0/4089959049 conn(0x7fd80c0f63b0 msgr2=0x7fd80c0f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:47:46.468 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:46.465+0000 7fd813c78700 1 -- 172.21.15.47:0/4089959049 shutdown_connections 2024-01-18T06:47:46.468 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:46.465+0000 7fd813c78700 1 -- 172.21.15.47:0/4089959049 wait complete. 2024-01-18T06:47:46.468 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:46.466+0000 7fd813c78700 1 Processor -- start 2024-01-18T06:47:46.468 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:46.466+0000 7fd813c78700 1 -- start start 2024-01-18T06:47:46.468 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:46.466+0000 7fd813c78700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd80c0fab90 0x7fd80c10aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:47:46.469 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:46.466+0000 7fd813c78700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd80c10aef0 con 0x7fd80c0fab90 2024-01-18T06:47:46.469 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:46.466+0000 7fd811a14700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd80c0fab90 0x7fd80c10aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:47:46.469 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:46.466+0000 7fd811a14700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd80c0fab90 0x7fd80c10aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52402/0 (socket says 172.21.15.47:52402) 2024-01-18T06:47:46.469 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:46.466+0000 7fd811a14700 1 -- 172.21.15.47:0/3639768822 learned_addr learned my addr 172.21.15.47:0/3639768822 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:47:46.469 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:46.467+0000 7fd811a14700 1 -- 172.21.15.47:0/3639768822 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd80800b200 con 0x7fd80c0fab90 2024-01-18T06:47:46.469 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:46.467+0000 7fd811a14700 1 --2- 172.21.15.47:0/3639768822 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd80c0fab90 0x7fd80c10aa00 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fd80800e040 tx=0x7fd808006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:47:46.469 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:46.467+0000 7fd7fa7fc700 1 -- 172.21.15.47:0/3639768822 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd80801b410 con 0x7fd80c0fab90 2024-01-18T06:47:46.469 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:46.467+0000 7fd7fa7fc700 1 -- 172.21.15.47:0/3639768822 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd80801b9e0 con 0x7fd80c0fab90 2024-01-18T06:47:46.469 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:46.467+0000 7fd813c78700 1 -- 172.21.15.47:0/3639768822 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd80c10b0f0 con 0x7fd80c0fab90 2024-01-18T06:47:46.469 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:46.467+0000 7fd7fa7fc700 1 -- 172.21.15.47:0/3639768822 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd80801a820 con 0x7fd80c0fab90 2024-01-18T06:47:46.469 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:46.467+0000 7fd813c78700 1 -- 172.21.15.47:0/3639768822 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd80c10b500 con 0x7fd80c0fab90 2024-01-18T06:47:46.469 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:46.468+0000 7fd7fa7fc700 1 -- 172.21.15.47:0/3639768822 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 3) v1 ==== 36029+0+0 (secure 0 0 0) 0x7fd80800f070 con 0x7fd80c0fab90 2024-01-18T06:47:46.469 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:46.468+0000 7fd7fa7fc700 1 --2- 172.21.15.47:0/3639768822 >> [v2:172.21.15.47:6800/3483218802,v1:172.21.15.47:6801/3483218802] conn(0x7fd7f402fca0 0x7fd7f4032160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:47:46.469 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:46.468+0000 7fd7fa7fc700 1 -- 172.21.15.47:0/3639768822 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fd808041160 con 0x7fd80c0fab90 2024-01-18T06:47:46.469 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:46.468+0000 7fd813c78700 1 -- 172.21.15.47:0/3639768822 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd80c045ad0 con 0x7fd80c0fab90 2024-01-18T06:47:46.470 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:46.468+0000 7fd811213700 1 --2- 172.21.15.47:0/3639768822 >> [v2:172.21.15.47:6800/3483218802,v1:172.21.15.47:6801/3483218802] conn(0x7fd7f402fca0 0x7fd7f4032160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:47:46.470 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:46.468+0000 7fd811213700 1 --2- 172.21.15.47:0/3639768822 >> [v2:172.21.15.47:6800/3483218802,v1:172.21.15.47:6801/3483218802] conn(0x7fd7f402fca0 0x7fd7f4032160 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fd7fc0096a0 tx=0x7fd7fc006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:47:46.472 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:46.472+0000 7fd7fa7fc700 1 -- 172.21.15.47:0/3639768822 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd808016020 con 0x7fd80c0fab90 2024-01-18T06:47:46.655 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:46.656+0000 7fd7fa7fc700 1 -- 172.21.15.47:0/3639768822 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7fd808023360 con 0x7fd80c0fab90 2024-01-18T06:47:46.669 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:46.669+0000 7fd813c78700 1 -- 172.21.15.47:0/3639768822 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fd80c1049c0 con 0x7fd80c0fab90 2024-01-18T06:47:46.963 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:46 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:47:45.656696+0000 mon.smithi047 (mon.0) 22 : cluster [DBG] mgrmap e3: smithi047.fuhzzr(active, since 1.00531s) 2024-01-18T06:47:46.963 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:46 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:47:45.695821+0000 mon.smithi047 (mon.0) 23 : audit [DBG] from='client.? 172.21.15.47:0/3172033933' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-18T06:47:47.657 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:47.657+0000 7fd7fa7fc700 1 -- 172.21.15.47:0/3639768822 <== mon.0 v2:172.21.15.47:3300/0 8 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fd8080430d0 con 0x7fd80c0fab90 2024-01-18T06:47:47.658 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:47.658+0000 7fd7fa7fc700 1 -- 172.21.15.47:0/3639768822 <== mon.0 v2:172.21.15.47:3300/0 9 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7fd8080109b0 con 0x7fd80c0fab90 2024-01-18T06:47:47.662 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:47.662+0000 7fd813c78700 1 -- 172.21.15.47:0/3639768822 >> [v2:172.21.15.47:6800/3483218802,v1:172.21.15.47:6801/3483218802] conn(0x7fd7f402fca0 msgr2=0x7fd7f4032160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:47:47.662 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:47.662+0000 7fd813c78700 1 --2- 172.21.15.47:0/3639768822 >> [v2:172.21.15.47:6800/3483218802,v1:172.21.15.47:6801/3483218802] conn(0x7fd7f402fca0 0x7fd7f4032160 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fd7fc0096a0 tx=0x7fd7fc006b40).stop 2024-01-18T06:47:47.662 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:47.662+0000 7fd813c78700 1 -- 172.21.15.47:0/3639768822 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd80c0fab90 msgr2=0x7fd80c10aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:47:47.662 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:47.662+0000 7fd813c78700 1 --2- 172.21.15.47:0/3639768822 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd80c0fab90 0x7fd80c10aa00 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fd80800e040 tx=0x7fd808006ee0).stop 2024-01-18T06:47:47.662 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:47.662+0000 7fd813c78700 1 -- 172.21.15.47:0/3639768822 shutdown_connections 2024-01-18T06:47:47.662 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:47.662+0000 7fd813c78700 1 --2- 172.21.15.47:0/3639768822 >> [v2:172.21.15.47:6800/3483218802,v1:172.21.15.47:6801/3483218802] conn(0x7fd7f402fca0 0x7fd7f4032160 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:47:47.662 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:47.662+0000 7fd813c78700 1 --2- 172.21.15.47:0/3639768822 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd80c0fab90 0x7fd80c10aa00 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:47:47.662 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:47.662+0000 7fd813c78700 1 -- 172.21.15.47:0/3639768822 >> 172.21.15.47:0/3639768822 conn(0x7fd80c0f63b0 msgr2=0x7fd80c0f6fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:47:47.663 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:47.662+0000 7fd813c78700 1 -- 172.21.15.47:0/3639768822 shutdown_connections 2024-01-18T06:47:47.663 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:47.662+0000 7fd813c78700 1 -- 172.21.15.47:0/3639768822 wait complete. 2024-01-18T06:47:47.963 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:47 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:47:46.657702+0000 mon.smithi047 (mon.0) 24 : cluster [DBG] mgrmap e4: smithi047.fuhzzr(active, since 2s) 2024-01-18T06:47:47.963 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:47 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:47:46.670573+0000 mon.smithi047 (mon.0) 25 : audit [INF] from='client.? 172.21.15.47:0/3639768822' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-01-18T06:47:48.067 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/653e4012-b5cd-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpck5s1jz7:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcg0aieo2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-01-18T06:47:48.443 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.443+0000 7f1685b02700 1 Processor -- start 2024-01-18T06:47:48.444 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.443+0000 7f1685b02700 1 -- start start 2024-01-18T06:47:48.444 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.443+0000 7f1685b02700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16800fab80 0x7f16800fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T06:47:48.444 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.443+0000 7f1685b02700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16800fb490 con 0x7f16800fab80 2024-01-18T06:47:48.444 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.443+0000 7f167f7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16800fab80 0x7f16800fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:47:48.444 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.443+0000 7f167f7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16800fab80 0x7f16800fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52432/0 (socket says 172.21.15.47:52432) 2024-01-18T06:47:48.444 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.443+0000 7f167f7fe700 1 -- 172.21.15.47:0/1351539728 learned_addr learned my addr 172.21.15.47:0/1351539728 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:47:48.444 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.444+0000 7f167f7fe700 1 -- 172.21.15.47:0/1351539728 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16800fb5d0 con 0x7f16800fab80 2024-01-18T06:47:48.444 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.444+0000 7f167f7fe700 1 --2- 172.21.15.47:0/1351539728 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16800fab80 0x7f16800fafa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f167000b5f0 tx=0x7f1670006ee0).ready entity=mon.0 client_cookie=3608e7bb2d28a96d server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:47:48.444 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.444+0000 7f167e7fc700 1 -- 172.21.15.47:0/1351539728 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f167000d600 con 0x7f16800fab80 2024-01-18T06:47:48.444 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.444+0000 7f167e7fc700 1 -- 172.21.15.47:0/1351539728 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f167000dbd0 con 0x7f16800fab80 2024-01-18T06:47:48.444 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.444+0000 7f167e7fc700 1 -- 172.21.15.47:0/1351539728 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1670011c00 con 0x7f16800fab80 2024-01-18T06:47:48.447 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.445+0000 7f1685b02700 1 -- 172.21.15.47:0/1351539728 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16800fab80 msgr2=0x7f16800fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:47:48.447 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.445+0000 7f1685b02700 1 --2- 172.21.15.47:0/1351539728 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16800fab80 0x7f16800fafa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f167000b5f0 tx=0x7f1670006ee0).stop 2024-01-18T06:47:48.447 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.445+0000 7f1685b02700 1 -- 172.21.15.47:0/1351539728 shutdown_connections 2024-01-18T06:47:48.447 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.445+0000 7f1685b02700 1 --2- 172.21.15.47:0/1351539728 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16800fab80 0x7f16800fafa0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:47:48.447 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.445+0000 7f1685b02700 1 -- 172.21.15.47:0/1351539728 >> 172.21.15.47:0/1351539728 conn(0x7f16800f63c0 msgr2=0x7f16800f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:47:48.447 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.445+0000 7f1685b02700 1 -- 172.21.15.47:0/1351539728 shutdown_connections 2024-01-18T06:47:48.447 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.445+0000 7f1685b02700 1 -- 172.21.15.47:0/1351539728 wait complete. 2024-01-18T06:47:48.447 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.446+0000 7f1685b02700 1 Processor -- start 2024-01-18T06:47:48.447 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.446+0000 7f1685b02700 1 -- start start 2024-01-18T06:47:48.447 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.446+0000 7f1685b02700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16800fab80 0x7f168010aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:47:48.447 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.446+0000 7f1685b02700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168010af10 con 0x7f16800fab80 2024-01-18T06:47:48.448 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.446+0000 7f167f7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16800fab80 0x7f168010aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:47:48.448 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.446+0000 7f167f7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16800fab80 0x7f168010aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52434/0 (socket says 172.21.15.47:52434) 2024-01-18T06:47:48.448 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.446+0000 7f167f7fe700 1 -- 172.21.15.47:0/1461605909 learned_addr learned my addr 172.21.15.47:0/1461605909 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:47:48.448 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.447+0000 7f167f7fe700 1 -- 172.21.15.47:0/1461605909 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f167000b200 con 0x7f16800fab80 2024-01-18T06:47:48.448 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.447+0000 7f167f7fe700 1 --2- 172.21.15.47:0/1461605909 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16800fab80 0x7f168010aa20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f167000e040 tx=0x7f1670006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:47:48.448 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.447+0000 7f1667fff700 1 -- 172.21.15.47:0/1461605909 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f167001b410 con 0x7f16800fab80 2024-01-18T06:47:48.448 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.447+0000 7f1667fff700 1 -- 172.21.15.47:0/1461605909 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f167001b9e0 con 0x7f16800fab80 2024-01-18T06:47:48.448 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.447+0000 7f1685b02700 1 -- 172.21.15.47:0/1461605909 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f168010b110 con 0x7f16800fab80 2024-01-18T06:47:48.448 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.447+0000 7f1667fff700 1 -- 172.21.15.47:0/1461605909 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f167001a820 con 0x7f16800fab80 2024-01-18T06:47:48.448 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.447+0000 7f1685b02700 1 -- 172.21.15.47:0/1461605909 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f168010b520 con 0x7f16800fab80 2024-01-18T06:47:48.449 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.448+0000 7f1667fff700 1 -- 172.21.15.47:0/1461605909 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f167000f070 con 0x7f16800fab80 2024-01-18T06:47:48.449 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.448+0000 7f1685b02700 1 -- 172.21.15.47:0/1461605909 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1680045ad0 con 0x7f16800fab80 2024-01-18T06:47:48.449 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.448+0000 7f1667fff700 1 --2- 172.21.15.47:0/1461605909 >> [v2:172.21.15.47:6800/3483218802,v1:172.21.15.47:6801/3483218802] conn(0x7f166c02fdb0 0x7f166c032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:47:48.449 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.448+0000 7f1667fff700 1 -- 172.21.15.47:0/1461605909 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f16700412d0 con 0x7f16800fab80 2024-01-18T06:47:48.449 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.448+0000 7f167effd700 1 -- 172.21.15.47:0/1461605909 >> [v2:172.21.15.47:6800/3483218802,v1:172.21.15.47:6801/3483218802] conn(0x7f166c02fdb0 msgr2=0x7f166c032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/3483218802 2024-01-18T06:47:48.449 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.448+0000 7f167effd700 1 --2- 172.21.15.47:0/1461605909 >> [v2:172.21.15.47:6800/3483218802,v1:172.21.15.47:6801/3483218802] conn(0x7f166c02fdb0 0x7f166c032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-18T06:47:48.452 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.453+0000 7f1667fff700 1 -- 172.21.15.47:0/1461605909 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f167001aea0 con 0x7f16800fab80 2024-01-18T06:47:48.648 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.649+0000 7f167effd700 1 -- 172.21.15.47:0/1461605909 >> [v2:172.21.15.47:6800/3483218802,v1:172.21.15.47:6801/3483218802] conn(0x7f166c02fdb0 msgr2=0x7f166c032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/3483218802 2024-01-18T06:47:48.649 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.649+0000 7f167effd700 1 --2- 172.21.15.47:0/1461605909 >> [v2:172.21.15.47:6800/3483218802,v1:172.21.15.47:6801/3483218802] conn(0x7f166c02fdb0 0x7f166c032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-18T06:47:48.652 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.653+0000 7f1685b02700 1 -- 172.21.15.47:0/1461605909 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f168010c630 con 0x7f16800fab80 2024-01-18T06:47:48.653 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.653+0000 7f1667fff700 1 -- 172.21.15.47:0/1461605909 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f1670016070 con 0x7f16800fab80 2024-01-18T06:47:48.653 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: { 2024-01-18T06:47:48.653 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 5, 2024-01-18T06:47:48.654 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "available": true, 2024-01-18T06:47:48.654 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "active_name": "smithi047.fuhzzr", 2024-01-18T06:47:48.654 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_standby": 0 2024-01-18T06:47:48.654 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: } 2024-01-18T06:47:48.655 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.655+0000 7f1685b02700 1 -- 172.21.15.47:0/1461605909 >> [v2:172.21.15.47:6800/3483218802,v1:172.21.15.47:6801/3483218802] conn(0x7f166c02fdb0 msgr2=0x7f166c032270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-18T06:47:48.655 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.655+0000 7f1685b02700 1 --2- 172.21.15.47:0/1461605909 >> [v2:172.21.15.47:6800/3483218802,v1:172.21.15.47:6801/3483218802] conn(0x7f166c02fdb0 0x7f166c032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-18T06:47:48.655 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.655+0000 7f1685b02700 1 -- 172.21.15.47:0/1461605909 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16800fab80 msgr2=0x7f168010aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:47:48.655 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.655+0000 7f1685b02700 1 --2- 172.21.15.47:0/1461605909 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16800fab80 0x7f168010aa20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f167000e040 tx=0x7f1670006ee0).stop 2024-01-18T06:47:48.655 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.655+0000 7f1685b02700 1 -- 172.21.15.47:0/1461605909 shutdown_connections 2024-01-18T06:47:48.655 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.655+0000 7f1685b02700 1 --2- 172.21.15.47:0/1461605909 >> [v2:172.21.15.47:6800/3483218802,v1:172.21.15.47:6801/3483218802] conn(0x7f166c02fdb0 0x7f166c032270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-18T06:47:48.656 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.655+0000 7f1685b02700 1 --2- 172.21.15.47:0/1461605909 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16800fab80 0x7f168010aa20 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:47:48.656 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.655+0000 7f1685b02700 1 -- 172.21.15.47:0/1461605909 >> 172.21.15.47:0/1461605909 conn(0x7f16800f63c0 msgr2=0x7f1680070030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:47:48.656 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.655+0000 7f1685b02700 1 -- 172.21.15.47:0/1461605909 shutdown_connections 2024-01-18T06:47:48.656 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:48.655+0000 7f1685b02700 1 -- 172.21.15.47:0/1461605909 wait complete. 2024-01-18T06:47:48.963 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:48 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:47:47.659226+0000 mon.smithi047 (mon.0) 26 : audit [INF] from='client.? 172.21.15.47:0/3639768822' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-01-18T06:47:48.963 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:48 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:47:47.659284+0000 mon.smithi047 (mon.0) 27 : cluster [DBG] mgrmap e5: smithi047.fuhzzr(active, since 3s) 2024-01-18T06:47:48.963 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:48 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:47:48.654282+0000 mon.smithi047 (mon.0) 28 : audit [DBG] from='client.? 172.21.15.47:0/1461605909' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-18T06:47:48.996 INFO:teuthology.orchestra.run.smithi047.stderr:Waiting for the mgr to restart... 2024-01-18T06:47:48.996 INFO:teuthology.orchestra.run.smithi047.stderr:Waiting for mgr epoch 5... 2024-01-18T06:47:48.996 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/653e4012-b5cd-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpck5s1jz7:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcg0aieo2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-01-18T06:47:49.357 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:49.355+0000 7f3b034ec700 1 Processor -- start 2024-01-18T06:47:49.357 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:49.356+0000 7f3b034ec700 1 -- start start 2024-01-18T06:47:49.357 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:49.356+0000 7f3b034ec700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3afc0fab80 0x7f3afc0fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T06:47:49.357 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:49.356+0000 7f3b034ec700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3afc0fb490 con 0x7f3afc0fab80 2024-01-18T06:47:49.357 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:49.356+0000 7f3b01288700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3afc0fab80 0x7f3afc0fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:47:49.358 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:49.356+0000 7f3b01288700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3afc0fab80 0x7f3afc0fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52446/0 (socket says 172.21.15.47:52446) 2024-01-18T06:47:49.358 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:49.356+0000 7f3b01288700 1 -- 172.21.15.47:0/3545670748 learned_addr learned my addr 172.21.15.47:0/3545670748 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:47:49.358 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:49.356+0000 7f3b01288700 1 -- 172.21.15.47:0/3545670748 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3afc0fb5d0 con 0x7f3afc0fab80 2024-01-18T06:47:49.358 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:49.356+0000 7f3b01288700 1 --2- 172.21.15.47:0/3545670748 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3afc0fab80 0x7f3afc0fafa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f3af800b870 tx=0x7f3af8006f70).ready entity=mon.0 client_cookie=47b6e156d02db6a9 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:47:49.358 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:49.357+0000 7f3aebfff700 1 -- 172.21.15.47:0/3545670748 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3af800da20 con 0x7f3afc0fab80 2024-01-18T06:47:49.358 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:49.357+0000 7f3aebfff700 1 -- 172.21.15.47:0/3545670748 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f3af8010470 con 0x7f3afc0fab80 2024-01-18T06:47:49.358 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:49.357+0000 7f3aebfff700 1 -- 172.21.15.47:0/3545670748 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3af800f5b0 con 0x7f3afc0fab80 2024-01-18T06:47:49.358 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:49.357+0000 7f3b034ec700 1 -- 172.21.15.47:0/3545670748 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3afc0fab80 msgr2=0x7f3afc0fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:47:49.358 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:49.357+0000 7f3b034ec700 1 --2- 172.21.15.47:0/3545670748 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3afc0fab80 0x7f3afc0fafa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f3af800b870 tx=0x7f3af8006f70).stop 2024-01-18T06:47:49.358 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:49.357+0000 7f3b034ec700 1 -- 172.21.15.47:0/3545670748 shutdown_connections 2024-01-18T06:47:49.358 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:49.357+0000 7f3b034ec700 1 --2- 172.21.15.47:0/3545670748 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3afc0fab80 0x7f3afc0fafa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:47:49.358 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:49.357+0000 7f3b034ec700 1 -- 172.21.15.47:0/3545670748 >> 172.21.15.47:0/3545670748 conn(0x7f3afc0f63c0 msgr2=0x7f3afc0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:47:49.358 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:49.357+0000 7f3b034ec700 1 -- 172.21.15.47:0/3545670748 shutdown_connections 2024-01-18T06:47:49.358 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:49.357+0000 7f3b034ec700 1 -- 172.21.15.47:0/3545670748 wait complete. 2024-01-18T06:47:49.358 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:49.358+0000 7f3b034ec700 1 Processor -- start 2024-01-18T06:47:49.359 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:49.358+0000 7f3b034ec700 1 -- start start 2024-01-18T06:47:49.361 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:49.358+0000 7f3b034ec700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3afc0fab80 0x7f3afc10cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:47:49.361 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:49.358+0000 7f3b034ec700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3afc10d090 con 0x7f3afc0fab80 2024-01-18T06:47:49.361 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:49.358+0000 7f3b01288700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3afc0fab80 0x7f3afc10cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:47:49.361 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:49.358+0000 7f3b01288700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3afc0fab80 0x7f3afc10cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:52450/0 (socket says 172.21.15.47:52450) 2024-01-18T06:47:49.361 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:49.358+0000 7f3b01288700 1 -- 172.21.15.47:0/920913626 learned_addr learned my addr 172.21.15.47:0/920913626 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:47:49.361 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:49.359+0000 7f3b01288700 1 -- 172.21.15.47:0/920913626 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af800b480 con 0x7f3afc0fab80 2024-01-18T06:47:49.361 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:49.359+0000 7f3b01288700 1 --2- 172.21.15.47:0/920913626 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3afc0fab80 0x7f3afc10cba0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f3af8006800 tx=0x7f3af8006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:47:49.361 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:49.359+0000 7f3ae9ffb700 1 -- 172.21.15.47:0/920913626 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3af800f980 con 0x7f3afc0fab80 2024-01-18T06:47:49.361 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:49.359+0000 7f3b034ec700 1 -- 172.21.15.47:0/920913626 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3afc10d290 con 0x7f3afc0fab80 2024-01-18T06:47:49.361 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:49.359+0000 7f3ae9ffb700 1 -- 172.21.15.47:0/920913626 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f3af8019450 con 0x7f3afc0fab80 2024-01-18T06:47:49.361 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:49.359+0000 7f3ae9ffb700 1 -- 172.21.15.47:0/920913626 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3af8023640 con 0x7f3afc0fab80 2024-01-18T06:47:49.361 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:49.359+0000 7f3b034ec700 1 -- 172.21.15.47:0/920913626 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3afc10d6a0 con 0x7f3afc0fab80 2024-01-18T06:47:49.361 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:49.360+0000 7f3ae9ffb700 1 -- 172.21.15.47:0/920913626 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f3af8016030 con 0x7f3afc0fab80 2024-01-18T06:47:49.362 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:49.360+0000 7f3ae9ffb700 1 --2- 172.21.15.47:0/920913626 >> [v2:172.21.15.47:6800/3483218802,v1:172.21.15.47:6801/3483218802] conn(0x7f3ae402fdb0 0x7f3ae4032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:47:49.362 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:49.360+0000 7f3ae9ffb700 1 -- 172.21.15.47:0/920913626 --> [v2:172.21.15.47:6800/3483218802,v1:172.21.15.47:6801/3483218802] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3ae4032930 con 0x7f3ae402fdb0 2024-01-18T06:47:49.362 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:49.360+0000 7f3ae9ffb700 1 -- 172.21.15.47:0/920913626 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f3af8041730 con 0x7f3afc0fab80 2024-01-18T06:47:49.362 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:49.360+0000 7f3b00a87700 1 -- 172.21.15.47:0/920913626 >> [v2:172.21.15.47:6800/3483218802,v1:172.21.15.47:6801/3483218802] conn(0x7f3ae402fdb0 msgr2=0x7f3ae4032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/3483218802 2024-01-18T06:47:49.362 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:49.360+0000 7f3b00a87700 1 --2- 172.21.15.47:0/920913626 >> [v2:172.21.15.47:6800/3483218802,v1:172.21.15.47:6801/3483218802] conn(0x7f3ae402fdb0 0x7f3ae4032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-18T06:47:49.561 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:49.561+0000 7f3b00a87700 1 -- 172.21.15.47:0/920913626 >> [v2:172.21.15.47:6800/3483218802,v1:172.21.15.47:6801/3483218802] conn(0x7f3ae402fdb0 msgr2=0x7f3ae4032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/3483218802 2024-01-18T06:47:49.561 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:49.561+0000 7f3b00a87700 1 --2- 172.21.15.47:0/920913626 >> [v2:172.21.15.47:6800/3483218802,v1:172.21.15.47:6801/3483218802] conn(0x7f3ae402fdb0 0x7f3ae4032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-18T06:47:49.961 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:49.962+0000 7f3b00a87700 1 -- 172.21.15.47:0/920913626 >> [v2:172.21.15.47:6800/3483218802,v1:172.21.15.47:6801/3483218802] conn(0x7f3ae402fdb0 msgr2=0x7f3ae4032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/3483218802 2024-01-18T06:47:49.962 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:49.962+0000 7f3b00a87700 1 --2- 172.21.15.47:0/920913626 >> [v2:172.21.15.47:6800/3483218802,v1:172.21.15.47:6801/3483218802] conn(0x7f3ae402fdb0 0x7f3ae4032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-01-18T06:47:50.762 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:50.763+0000 7f3b00a87700 1 -- 172.21.15.47:0/920913626 >> [v2:172.21.15.47:6800/3483218802,v1:172.21.15.47:6801/3483218802] conn(0x7f3ae402fdb0 msgr2=0x7f3ae4032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/3483218802 2024-01-18T06:47:50.763 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:50.763+0000 7f3b00a87700 1 --2- 172.21.15.47:0/920913626 >> [v2:172.21.15.47:6800/3483218802,v1:172.21.15.47:6801/3483218802] conn(0x7f3ae402fdb0 0x7f3ae4032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-01-18T06:47:52.364 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:52.364+0000 7f3b00a87700 1 -- 172.21.15.47:0/920913626 >> [v2:172.21.15.47:6800/3483218802,v1:172.21.15.47:6801/3483218802] conn(0x7f3ae402fdb0 msgr2=0x7f3ae4032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/3483218802 2024-01-18T06:47:52.364 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:52.364+0000 7f3b00a87700 1 --2- 172.21.15.47:0/920913626 >> [v2:172.21.15.47:6800/3483218802,v1:172.21.15.47:6801/3483218802] conn(0x7f3ae402fdb0 0x7f3ae4032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-01-18T06:47:52.874 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:52.875+0000 7f3ae9ffb700 1 -- 172.21.15.47:0/920913626 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mgrmap(e 6) v1 ==== 35970+0+0 (secure 0 0 0) 0x7f3af80237a0 con 0x7f3afc0fab80 2024-01-18T06:47:52.875 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:52.875+0000 7f3ae9ffb700 1 -- 172.21.15.47:0/920913626 >> [v2:172.21.15.47:6800/3483218802,v1:172.21.15.47:6801/3483218802] conn(0x7f3ae402fdb0 msgr2=0x7f3ae4032270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-18T06:47:52.875 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:52.875+0000 7f3ae9ffb700 1 --2- 172.21.15.47:0/920913626 >> [v2:172.21.15.47:6800/3483218802,v1:172.21.15.47:6801/3483218802] conn(0x7f3ae402fdb0 0x7f3ae4032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-18T06:47:53.876 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:53.876+0000 7f3ae9ffb700 1 -- 172.21.15.47:0/920913626 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f3af8023ad0 con 0x7f3afc0fab80 2024-01-18T06:47:53.876 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:53.876+0000 7f3ae9ffb700 1 --2- 172.21.15.47:0/920913626 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f3ae4033880 0x7f3ae4035c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:47:53.876 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:53.876+0000 7f3ae9ffb700 1 -- 172.21.15.47:0/920913626 --> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3ae4032930 con 0x7f3ae4033880 2024-01-18T06:47:53.877 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:53.877+0000 7f3b00a87700 1 --2- 172.21.15.47:0/920913626 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f3ae4033880 0x7f3ae4035c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:47:53.877 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:53.877+0000 7f3b00a87700 1 --2- 172.21.15.47:0/920913626 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f3ae4033880 0x7f3ae4035c70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f3aec006fd0 tx=0x7f3aec006d70).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:47:53.878 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:53.878+0000 7f3ae9ffb700 1 -- 172.21.15.47:0/920913626 <== mgr.14114 v2:172.21.15.47:6800/3978480908 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f3ae4032930 con 0x7f3ae4033880 2024-01-18T06:47:53.884 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:53.885+0000 7f3b034ec700 1 -- 172.21.15.47:0/920913626 --> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f3afc059100 con 0x7f3ae4033880 2024-01-18T06:47:53.884 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:53.885+0000 7f3ae9ffb700 1 -- 172.21.15.47:0/920913626 <== mgr.14114 v2:172.21.15.47:6800/3978480908 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f3afc059100 con 0x7f3ae4033880 2024-01-18T06:47:53.884 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: { 2024-01-18T06:47:53.885 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-01-18T06:47:53.885 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "initialized": true 2024-01-18T06:47:53.885 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: } 2024-01-18T06:47:53.885 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:53.885+0000 7f3b034ec700 1 -- 172.21.15.47:0/920913626 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f3ae4033880 msgr2=0x7f3ae4035c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:47:53.886 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:53.885+0000 7f3b034ec700 1 --2- 172.21.15.47:0/920913626 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f3ae4033880 0x7f3ae4035c70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f3aec006fd0 tx=0x7f3aec006d70).stop 2024-01-18T06:47:53.886 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:53.885+0000 7f3b034ec700 1 -- 172.21.15.47:0/920913626 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3afc0fab80 msgr2=0x7f3afc10cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:47:53.886 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:53.885+0000 7f3b034ec700 1 --2- 172.21.15.47:0/920913626 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3afc0fab80 0x7f3afc10cba0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f3af8006800 tx=0x7f3af8006f70).stop 2024-01-18T06:47:53.886 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:53.885+0000 7f3b034ec700 1 -- 172.21.15.47:0/920913626 shutdown_connections 2024-01-18T06:47:53.887 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:53.885+0000 7f3b034ec700 1 --2- 172.21.15.47:0/920913626 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f3ae4033880 0x7f3ae4035c70 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:47:53.887 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:53.885+0000 7f3b034ec700 1 --2- 172.21.15.47:0/920913626 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3afc0fab80 0x7f3afc10cba0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:47:53.887 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:53.885+0000 7f3b034ec700 1 --2- 172.21.15.47:0/920913626 >> [v2:172.21.15.47:6800/3483218802,v1:172.21.15.47:6801/3483218802] conn(0x7f3ae402fdb0 0x7f3ae4032270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-18T06:47:53.887 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:53.885+0000 7f3b034ec700 1 -- 172.21.15.47:0/920913626 >> 172.21.15.47:0/920913626 conn(0x7f3afc0f63c0 msgr2=0x7f3afc103d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:47:53.887 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:53.886+0000 7f3b034ec700 1 -- 172.21.15.47:0/920913626 shutdown_connections 2024-01-18T06:47:53.887 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:53.886+0000 7f3b034ec700 1 -- 172.21.15.47:0/920913626 wait complete. 2024-01-18T06:47:54.090 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:53 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:47:52.823391+0000 mon.smithi047 (mon.0) 29 : cluster [INF] Active manager daemon smithi047.fuhzzr restarted 2024-01-18T06:47:54.090 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:53 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:47:52.823768+0000 mon.smithi047 (mon.0) 30 : cluster [INF] Activating manager daemon smithi047.fuhzzr 2024-01-18T06:47:54.090 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:53 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:47:52.825357+0000 mon.smithi047 (mon.0) 31 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-01-18T06:47:54.090 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:53 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:47:52.876285+0000 mon.smithi047 (mon.0) 32 : cluster [DBG] mgrmap e6: smithi047.fuhzzr(active, starting, since 0.0525883s) 2024-01-18T06:47:54.090 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:53 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:47:52.877206+0000 mon.smithi047 (mon.0) 33 : audit [DBG] from='mgr.14114 172.21.15.47:0/2934400552' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix": "mon metadata", "id": "smithi047"}]: dispatch 2024-01-18T06:47:54.091 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:53 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:47:52.877882+0000 mon.smithi047 (mon.0) 34 : audit [DBG] from='mgr.14114 172.21.15.47:0/2934400552' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix": "mgr metadata", "who": "smithi047.fuhzzr", "id": "smithi047.fuhzzr"}]: dispatch 2024-01-18T06:47:54.091 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:53 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:47:52.878561+0000 mon.smithi047 (mon.0) 35 : audit [DBG] from='mgr.14114 172.21.15.47:0/2934400552' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-18T06:47:54.091 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:53 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:47:52.878691+0000 mon.smithi047 (mon.0) 36 : audit [DBG] from='mgr.14114 172.21.15.47:0/2934400552' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-18T06:47:54.091 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:53 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:47:52.878819+0000 mon.smithi047 (mon.0) 37 : audit [DBG] from='mgr.14114 172.21.15.47:0/2934400552' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-18T06:47:54.091 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:53 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:47:52.892054+0000 mon.smithi047 (mon.0) 38 : cluster [INF] Manager daemon smithi047.fuhzzr is now available 2024-01-18T06:47:54.091 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:53 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:47:52.906388+0000 mon.smithi047 (mon.0) 39 : audit [INF] from='mgr.14114 172.21.15.47:0/2934400552' entity='mgr.smithi047.fuhzzr' 2024-01-18T06:47:54.091 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:53 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:47:52.909274+0000 mon.smithi047 (mon.0) 40 : audit [INF] from='mgr.14114 172.21.15.47:0/2934400552' entity='mgr.smithi047.fuhzzr' 2024-01-18T06:47:54.091 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:53 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:47:52.911789+0000 mon.smithi047 (mon.0) 41 : audit [INF] from='mgr.14114 172.21.15.47:0/2934400552' entity='mgr.smithi047.fuhzzr' 2024-01-18T06:47:54.091 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:53 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:47:53.005370+0000 mon.smithi047 (mon.0) 42 : audit [INF] from='mgr.14114 172.21.15.47:0/2934400552' entity='mgr.smithi047.fuhzzr' 2024-01-18T06:47:54.091 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:53 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:47:53.006045+0000 mon.smithi047 (mon.0) 43 : audit [DBG] from='mgr.14114 172.21.15.47:0/2934400552' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-18T06:47:54.091 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:53 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:47:53.007802+0000 mon.smithi047 (mon.0) 44 : audit [DBG] from='mgr.14114 172.21.15.47:0/2934400552' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-18T06:47:54.091 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:53 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:47:53.009270+0000 mon.smithi047 (mon.0) 45 : audit [DBG] from='mgr.14114 172.21.15.47:0/2934400552' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-18T06:47:54.091 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:53 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:47:53.019849+0000 mon.smithi047 (mon.0) 46 : audit [INF] from='mgr.14114 172.21.15.47:0/2934400552' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.fuhzzr/mirror_snapshot_schedule"}]: dispatch 2024-01-18T06:47:54.091 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:53 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:47:53.021031+0000 mon.smithi047 (mon.0) 47 : audit [INF] from='mgr.14114 172.21.15.47:0/2934400552' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.fuhzzr/trash_purge_schedule"}]: dispatch 2024-01-18T06:47:54.241 INFO:teuthology.orchestra.run.smithi047.stderr:mgr epoch 5 is available 2024-01-18T06:47:54.242 INFO:teuthology.orchestra.run.smithi047.stderr:Setting orchestrator backend to cephadm... 2024-01-18T06:47:54.242 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/653e4012-b5cd-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpck5s1jz7:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcg0aieo2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2024-01-18T06:47:54.642 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.640+0000 7f8a72e07700 1 Processor -- start 2024-01-18T06:47:54.642 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.640+0000 7f8a72e07700 1 -- start start 2024-01-18T06:47:54.642 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.640+0000 7f8a72e07700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a6c0703f0 0x7f8a6c0fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T06:47:54.642 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.640+0000 7f8a72e07700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a6c0fee10 con 0x7f8a6c0703f0 2024-01-18T06:47:54.642 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.640+0000 7f8a70ba3700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a6c0703f0 0x7f8a6c0fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:47:54.642 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.641+0000 7f8a70ba3700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a6c0703f0 0x7f8a6c0fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33366/0 (socket says 172.21.15.47:33366) 2024-01-18T06:47:54.642 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.641+0000 7f8a70ba3700 1 -- 172.21.15.47:0/954675464 learned_addr learned my addr 172.21.15.47:0/954675464 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:47:54.642 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.641+0000 7f8a70ba3700 1 -- 172.21.15.47:0/954675464 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a6c0fef50 con 0x7f8a6c0703f0 2024-01-18T06:47:54.643 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.641+0000 7f8a70ba3700 1 --2- 172.21.15.47:0/954675464 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a6c0703f0 0x7f8a6c0fe920 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f8a6000b5f0 tx=0x7f8a60006ee0).ready entity=mon.0 client_cookie=996aa9b9edb1a6ad server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:47:54.643 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.641+0000 7f8a6b7fe700 1 -- 172.21.15.47:0/954675464 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a6000d600 con 0x7f8a6c0703f0 2024-01-18T06:47:54.643 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.641+0000 7f8a6b7fe700 1 -- 172.21.15.47:0/954675464 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8a6000dbd0 con 0x7f8a6c0703f0 2024-01-18T06:47:54.643 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.641+0000 7f8a6b7fe700 1 -- 172.21.15.47:0/954675464 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a60011c00 con 0x7f8a6c0703f0 2024-01-18T06:47:54.643 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.642+0000 7f8a72e07700 1 -- 172.21.15.47:0/954675464 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a6c0703f0 msgr2=0x7f8a6c0fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:47:54.643 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.642+0000 7f8a72e07700 1 --2- 172.21.15.47:0/954675464 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a6c0703f0 0x7f8a6c0fe920 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f8a6000b5f0 tx=0x7f8a60006ee0).stop 2024-01-18T06:47:54.643 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.642+0000 7f8a72e07700 1 -- 172.21.15.47:0/954675464 shutdown_connections 2024-01-18T06:47:54.643 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.642+0000 7f8a72e07700 1 --2- 172.21.15.47:0/954675464 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a6c0703f0 0x7f8a6c0fe920 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:47:54.643 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.642+0000 7f8a72e07700 1 -- 172.21.15.47:0/954675464 >> 172.21.15.47:0/954675464 conn(0x7f8a6c0f63c0 msgr2=0x7f8a6c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:47:54.643 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.642+0000 7f8a72e07700 1 -- 172.21.15.47:0/954675464 shutdown_connections 2024-01-18T06:47:54.643 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.642+0000 7f8a72e07700 1 -- 172.21.15.47:0/954675464 wait complete. 2024-01-18T06:47:54.643 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.642+0000 7f8a72e07700 1 Processor -- start 2024-01-18T06:47:54.643 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.642+0000 7f8a72e07700 1 -- start start 2024-01-18T06:47:54.643 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.643+0000 7f8a72e07700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a6c0703f0 0x7f8a6c10aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:47:54.643 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.643+0000 7f8a72e07700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a6c10af10 con 0x7f8a6c0703f0 2024-01-18T06:47:54.645 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.643+0000 7f8a70ba3700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a6c0703f0 0x7f8a6c10aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:47:54.645 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.643+0000 7f8a70ba3700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a6c0703f0 0x7f8a6c10aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33382/0 (socket says 172.21.15.47:33382) 2024-01-18T06:47:54.645 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.643+0000 7f8a70ba3700 1 -- 172.21.15.47:0/2268176964 learned_addr learned my addr 172.21.15.47:0/2268176964 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:47:54.645 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.643+0000 7f8a70ba3700 1 -- 172.21.15.47:0/2268176964 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a6000b200 con 0x7f8a6c0703f0 2024-01-18T06:47:54.646 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.643+0000 7f8a70ba3700 1 --2- 172.21.15.47:0/2268176964 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a6c0703f0 0x7f8a6c10aa20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f8a6000e040 tx=0x7f8a60006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:47:54.646 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.643+0000 7f8a697fa700 1 -- 172.21.15.47:0/2268176964 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a6000d9f0 con 0x7f8a6c0703f0 2024-01-18T06:47:54.646 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.644+0000 7f8a697fa700 1 -- 172.21.15.47:0/2268176964 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8a6001a9f0 con 0x7f8a6c0703f0 2024-01-18T06:47:54.646 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.644+0000 7f8a72e07700 1 -- 172.21.15.47:0/2268176964 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a6c10b110 con 0x7f8a6c0703f0 2024-01-18T06:47:54.646 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.644+0000 7f8a697fa700 1 -- 172.21.15.47:0/2268176964 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a6001b8c0 con 0x7f8a6c0703f0 2024-01-18T06:47:54.646 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.644+0000 7f8a72e07700 1 -- 172.21.15.47:0/2268176964 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a6c10b520 con 0x7f8a6c0703f0 2024-01-18T06:47:54.646 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.644+0000 7f8a697fa700 1 -- 172.21.15.47:0/2268176964 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f8a6000f070 con 0x7f8a6c0703f0 2024-01-18T06:47:54.646 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.644+0000 7f8a697fa700 1 --2- 172.21.15.47:0/2268176964 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f8a5002fcc0 0x7f8a50032180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:47:54.646 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.644+0000 7f8a697fa700 1 -- 172.21.15.47:0/2268176964 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f8a60041200 con 0x7f8a6c0703f0 2024-01-18T06:47:54.646 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.645+0000 7f8a6bfff700 1 --2- 172.21.15.47:0/2268176964 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f8a5002fcc0 0x7f8a50032180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:47:54.646 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.645+0000 7f8a6bfff700 1 --2- 172.21.15.47:0/2268176964 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f8a5002fcc0 0x7f8a50032180 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f8a580096a0 tx=0x7f8a58006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:47:54.647 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.645+0000 7f8a72e07700 1 -- 172.21.15.47:0/2268176964 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a6c059100 con 0x7f8a6c0703f0 2024-01-18T06:47:54.649 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.649+0000 7f8a697fa700 1 -- 172.21.15.47:0/2268176964 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8a60016070 con 0x7f8a6c0703f0 2024-01-18T06:47:54.837 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.837+0000 7f8a72e07700 1 -- 172.21.15.47:0/2268176964 --> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f8a6c1048f0 con 0x7f8a5002fcc0 2024-01-18T06:47:54.842 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.842+0000 7f8a697fa700 1 -- 172.21.15.47:0/2268176964 <== mgr.14114 v2:172.21.15.47:6800/3978480908 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8a6c1048f0 con 0x7f8a5002fcc0 2024-01-18T06:47:54.844 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.844+0000 7f8a72e07700 1 -- 172.21.15.47:0/2268176964 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f8a5002fcc0 msgr2=0x7f8a50032180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:47:54.844 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.844+0000 7f8a72e07700 1 --2- 172.21.15.47:0/2268176964 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f8a5002fcc0 0x7f8a50032180 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f8a580096a0 tx=0x7f8a58006b40).stop 2024-01-18T06:47:54.845 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.844+0000 7f8a72e07700 1 -- 172.21.15.47:0/2268176964 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a6c0703f0 msgr2=0x7f8a6c10aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:47:54.845 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.844+0000 7f8a72e07700 1 --2- 172.21.15.47:0/2268176964 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a6c0703f0 0x7f8a6c10aa20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f8a6000e040 tx=0x7f8a60006ee0).stop 2024-01-18T06:47:54.845 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.844+0000 7f8a72e07700 1 -- 172.21.15.47:0/2268176964 shutdown_connections 2024-01-18T06:47:54.845 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.844+0000 7f8a72e07700 1 --2- 172.21.15.47:0/2268176964 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f8a5002fcc0 0x7f8a50032180 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:47:54.845 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.844+0000 7f8a72e07700 1 --2- 172.21.15.47:0/2268176964 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a6c0703f0 0x7f8a6c10aa20 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:47:54.845 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.844+0000 7f8a72e07700 1 -- 172.21.15.47:0/2268176964 >> 172.21.15.47:0/2268176964 conn(0x7f8a6c0f63c0 msgr2=0x7f8a6c0f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:47:54.845 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.844+0000 7f8a72e07700 1 -- 172.21.15.47:0/2268176964 shutdown_connections 2024-01-18T06:47:54.845 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:54.844+0000 7f8a72e07700 1 -- 172.21.15.47:0/2268176964 wait complete. 2024-01-18T06:47:55.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:54 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:47:53.878549+0000 mgr.smithi047.fuhzzr (mgr.14114) 1 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-18T06:47:55.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:54 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:47:53.879488+0000 mon.smithi047 (mon.0) 48 : cluster [DBG] mgrmap e7: smithi047.fuhzzr(active, since 1.05578s) 2024-01-18T06:47:55.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:54 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:47:53.885702+0000 mgr.smithi047.fuhzzr (mgr.14114) 2 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-18T06:47:55.214 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:54 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:47:54.842907+0000 mon.smithi047 (mon.0) 49 : audit [INF] from='mgr.14114 172.21.15.47:0/2934400552' entity='mgr.smithi047.fuhzzr' 2024-01-18T06:47:55.214 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:54 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:47:54.846708+0000 mon.smithi047 (mon.0) 50 : audit [DBG] from='mgr.14114 172.21.15.47:0/2934400552' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-18T06:47:55.963 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:55 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:47:54.839087+0000 mgr.smithi047.fuhzzr (mgr.14114) 3 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-01-18T06:47:55.964 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:55 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:47:54.880589+0000 mon.smithi047 (mon.0) 51 : cluster [DBG] mgrmap e8: smithi047.fuhzzr(active, since 2s) 2024-01-18T06:47:56.533 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/653e4012-b5cd-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpck5s1jz7:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcg0aieo2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2024-01-18T06:47:58.024 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.804+0000 7f4603a50700 1 Processor -- start 2024-01-18T06:47:58.024 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.804+0000 7f4603a50700 1 -- start start 2024-01-18T06:47:58.024 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.805+0000 7f4603a50700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45fc072490 0x7f45fc070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T06:47:58.024 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.805+0000 7f4603a50700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45fc0728b0 con 0x7f45fc072490 2024-01-18T06:47:58.024 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.805+0000 7f46017ec700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45fc072490 0x7f45fc070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:47:58.024 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.805+0000 7f46017ec700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45fc072490 0x7f45fc070a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33398/0 (socket says 172.21.15.47:33398) 2024-01-18T06:47:58.024 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.805+0000 7f46017ec700 1 -- 172.21.15.47:0/1200144190 learned_addr learned my addr 172.21.15.47:0/1200144190 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:47:58.024 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.805+0000 7f46017ec700 1 -- 172.21.15.47:0/1200144190 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45fc070f20 con 0x7f45fc072490 2024-01-18T06:47:58.024 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.805+0000 7f46017ec700 1 --2- 172.21.15.47:0/1200144190 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45fc072490 0x7f45fc070a30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f45ec00b5f0 tx=0x7f45ec006ee0).ready entity=mon.0 client_cookie=f0b87aa37c18e6d1 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:47:58.024 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.805+0000 7f45ebfff700 1 -- 172.21.15.47:0/1200144190 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45ec00d600 con 0x7f45fc072490 2024-01-18T06:47:58.025 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.806+0000 7f45ebfff700 1 -- 172.21.15.47:0/1200144190 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f45ec00dbd0 con 0x7f45fc072490 2024-01-18T06:47:58.025 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.806+0000 7f45ebfff700 1 -- 172.21.15.47:0/1200144190 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45ec011c00 con 0x7f45fc072490 2024-01-18T06:47:58.025 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.806+0000 7f4603a50700 1 -- 172.21.15.47:0/1200144190 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45fc072490 msgr2=0x7f45fc070a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:47:58.025 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.806+0000 7f4603a50700 1 --2- 172.21.15.47:0/1200144190 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45fc072490 0x7f45fc070a30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f45ec00b5f0 tx=0x7f45ec006ee0).stop 2024-01-18T06:47:58.025 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.806+0000 7f4603a50700 1 -- 172.21.15.47:0/1200144190 shutdown_connections 2024-01-18T06:47:58.025 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.806+0000 7f4603a50700 1 --2- 172.21.15.47:0/1200144190 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45fc072490 0x7f45fc070a30 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:47:58.025 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.806+0000 7f4603a50700 1 -- 172.21.15.47:0/1200144190 >> 172.21.15.47:0/1200144190 conn(0x7f45fc0f74c0 msgr2=0x7f45fc0f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:47:58.025 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.806+0000 7f4603a50700 1 -- 172.21.15.47:0/1200144190 shutdown_connections 2024-01-18T06:47:58.025 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.806+0000 7f4603a50700 1 -- 172.21.15.47:0/1200144190 wait complete. 2024-01-18T06:47:58.025 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.807+0000 7f4603a50700 1 Processor -- start 2024-01-18T06:47:58.025 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.807+0000 7f4603a50700 1 -- start start 2024-01-18T06:47:58.025 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.807+0000 7f4603a50700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45fc072490 0x7f45fc10cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:47:58.025 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.807+0000 7f4603a50700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45fc10d1f0 con 0x7f45fc072490 2024-01-18T06:47:58.025 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.807+0000 7f46017ec700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45fc072490 0x7f45fc10cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:47:58.026 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.807+0000 7f46017ec700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45fc072490 0x7f45fc10cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33412/0 (socket says 172.21.15.47:33412) 2024-01-18T06:47:58.026 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.807+0000 7f46017ec700 1 -- 172.21.15.47:0/3408959712 learned_addr learned my addr 172.21.15.47:0/3408959712 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:47:58.026 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.807+0000 7f46017ec700 1 -- 172.21.15.47:0/3408959712 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45ec00b200 con 0x7f45fc072490 2024-01-18T06:47:58.026 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.808+0000 7f46017ec700 1 --2- 172.21.15.47:0/3408959712 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45fc072490 0x7f45fc10cd00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f45ec011820 tx=0x7f45ec006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:47:58.026 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.808+0000 7f45e9ffb700 1 -- 172.21.15.47:0/3408959712 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45ec01b410 con 0x7f45fc072490 2024-01-18T06:47:58.026 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.808+0000 7f45e9ffb700 1 -- 172.21.15.47:0/3408959712 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f45ec01b9e0 con 0x7f45fc072490 2024-01-18T06:47:58.026 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.808+0000 7f45e9ffb700 1 -- 172.21.15.47:0/3408959712 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45ec01a8b0 con 0x7f45fc072490 2024-01-18T06:47:58.026 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.808+0000 7f4603a50700 1 -- 172.21.15.47:0/3408959712 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45fc10d3f0 con 0x7f45fc072490 2024-01-18T06:47:58.026 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.808+0000 7f4603a50700 1 -- 172.21.15.47:0/3408959712 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45fc10d800 con 0x7f45fc072490 2024-01-18T06:47:58.026 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.809+0000 7f45e9ffb700 1 -- 172.21.15.47:0/3408959712 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f45ec00f070 con 0x7f45fc072490 2024-01-18T06:47:58.026 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.809+0000 7f45e9ffb700 1 --2- 172.21.15.47:0/3408959712 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f45e402fdb0 0x7f45e4032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:47:58.026 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.809+0000 7f45e9ffb700 1 -- 172.21.15.47:0/3408959712 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f45ec041300 con 0x7f45fc072490 2024-01-18T06:47:58.026 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.809+0000 7f4600feb700 1 --2- 172.21.15.47:0/3408959712 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f45e402fdb0 0x7f45e4032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:47:58.026 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.809+0000 7f4603a50700 1 -- 172.21.15.47:0/3408959712 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45fc045ad0 con 0x7f45fc072490 2024-01-18T06:47:58.027 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.809+0000 7f4600feb700 1 --2- 172.21.15.47:0/3408959712 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f45e402fdb0 0x7f45e4032270 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f45f80096a0 tx=0x7f45f8006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:47:58.027 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.813+0000 7f45e9ffb700 1 -- 172.21.15.47:0/3408959712 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f45ec016070 con 0x7f45fc072490 2024-01-18T06:47:58.027 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.976+0000 7f4603a50700 1 -- 172.21.15.47:0/3408959712 --> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f45fc10ded0 con 0x7f45e402fdb0 2024-01-18T06:47:58.027 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.976+0000 7f45e9ffb700 1 -- 172.21.15.47:0/3408959712 <== mgr.14114 v2:172.21.15.47:6800/3978480908 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f45fc10ded0 con 0x7f45e402fdb0 2024-01-18T06:47:58.027 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.978+0000 7f4603a50700 1 -- 172.21.15.47:0/3408959712 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f45e402fdb0 msgr2=0x7f45e4032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:47:58.027 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.978+0000 7f4603a50700 1 --2- 172.21.15.47:0/3408959712 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f45e402fdb0 0x7f45e4032270 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f45f80096a0 tx=0x7f45f8006b40).stop 2024-01-18T06:47:58.027 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.978+0000 7f4603a50700 1 -- 172.21.15.47:0/3408959712 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45fc072490 msgr2=0x7f45fc10cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:47:58.027 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.978+0000 7f4603a50700 1 --2- 172.21.15.47:0/3408959712 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45fc072490 0x7f45fc10cd00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f45ec011820 tx=0x7f45ec006ee0).stop 2024-01-18T06:47:58.027 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.978+0000 7f4603a50700 1 -- 172.21.15.47:0/3408959712 shutdown_connections 2024-01-18T06:47:58.027 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.978+0000 7f4603a50700 1 --2- 172.21.15.47:0/3408959712 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f45e402fdb0 0x7f45e4032270 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:47:58.027 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.978+0000 7f4603a50700 1 --2- 172.21.15.47:0/3408959712 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45fc072490 0x7f45fc10cd00 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:47:58.027 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.978+0000 7f4603a50700 1 -- 172.21.15.47:0/3408959712 >> 172.21.15.47:0/3408959712 conn(0x7f45fc0f74c0 msgr2=0x7f45fc0f8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:47:58.027 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.978+0000 7f4603a50700 1 -- 172.21.15.47:0/3408959712 shutdown_connections 2024-01-18T06:47:58.028 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:47:57.978+0000 7f4603a50700 1 -- 172.21.15.47:0/3408959712 wait complete. 2024-01-18T06:47:58.028 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: value unchanged 2024-01-18T06:47:59.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:47:58 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:47:57.977045+0000 mgr.smithi047.fuhzzr (mgr.14114) 4 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-01-18T06:47:59.741 INFO:teuthology.orchestra.run.smithi047.stderr:Generating ssh key... 2024-01-18T06:47:59.741 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/653e4012-b5cd-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpck5s1jz7:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcg0aieo2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2024-01-18T06:48:00.853 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:00.720+0000 7f4677899700 1 Processor -- start 2024-01-18T06:48:00.853 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:00.720+0000 7f4677899700 1 -- start start 2024-01-18T06:48:00.854 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:00.720+0000 7f4677899700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4670070430 0x7f4670100ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:00.854 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:00.720+0000 7f4677899700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46700fc6d0 con 0x7f4670070430 2024-01-18T06:48:00.854 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:00.720+0000 7f4675635700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4670070430 0x7f4670100ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:00.854 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:00.720+0000 7f4675635700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4670070430 0x7f4670100ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33420/0 (socket says 172.21.15.47:33420) 2024-01-18T06:48:00.854 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:00.720+0000 7f4675635700 1 -- 172.21.15.47:0/3088363915 learned_addr learned my addr 172.21.15.47:0/3088363915 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:48:00.854 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:00.721+0000 7f4675635700 1 -- 172.21.15.47:0/3088363915 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46701011d0 con 0x7f4670070430 2024-01-18T06:48:00.854 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:00.721+0000 7f4675635700 1 --2- 172.21.15.47:0/3088363915 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4670070430 0x7f4670100ce0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f466c00b5f0 tx=0x7f466c006ee0).ready entity=mon.0 client_cookie=f2d2244eb956b70c server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:00.854 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:00.721+0000 7f4663fff700 1 -- 172.21.15.47:0/3088363915 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f466c00d600 con 0x7f4670070430 2024-01-18T06:48:00.854 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:00.721+0000 7f4663fff700 1 -- 172.21.15.47:0/3088363915 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f466c00dbd0 con 0x7f4670070430 2024-01-18T06:48:00.854 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:00.721+0000 7f4663fff700 1 -- 172.21.15.47:0/3088363915 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f466c011c00 con 0x7f4670070430 2024-01-18T06:48:00.854 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:00.721+0000 7f4677899700 1 -- 172.21.15.47:0/3088363915 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4670070430 msgr2=0x7f4670100ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:00.854 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:00.721+0000 7f4677899700 1 --2- 172.21.15.47:0/3088363915 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4670070430 0x7f4670100ce0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f466c00b5f0 tx=0x7f466c006ee0).stop 2024-01-18T06:48:00.854 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:00.721+0000 7f4677899700 1 -- 172.21.15.47:0/3088363915 shutdown_connections 2024-01-18T06:48:00.854 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:00.721+0000 7f4677899700 1 --2- 172.21.15.47:0/3088363915 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4670070430 0x7f4670100ce0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:00.855 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:00.721+0000 7f4677899700 1 -- 172.21.15.47:0/3088363915 >> 172.21.15.47:0/3088363915 conn(0x7f46700f63c0 msgr2=0x7f46700f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:48:00.855 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:00.721+0000 7f4677899700 1 -- 172.21.15.47:0/3088363915 shutdown_connections 2024-01-18T06:48:00.855 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:00.722+0000 7f4677899700 1 -- 172.21.15.47:0/3088363915 wait complete. 2024-01-18T06:48:00.855 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:00.722+0000 7f4677899700 1 Processor -- start 2024-01-18T06:48:00.855 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:00.722+0000 7f4677899700 1 -- start start 2024-01-18T06:48:00.855 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:00.722+0000 7f4677899700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4670070430 0x7f467010cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:00.855 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:00.722+0000 7f4677899700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f467010d0c0 con 0x7f4670070430 2024-01-18T06:48:00.856 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:00.722+0000 7f4675635700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4670070430 0x7f467010cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:00.856 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:00.722+0000 7f4675635700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4670070430 0x7f467010cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33426/0 (socket says 172.21.15.47:33426) 2024-01-18T06:48:00.856 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:00.722+0000 7f4675635700 1 -- 172.21.15.47:0/1106936080 learned_addr learned my addr 172.21.15.47:0/1106936080 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:48:00.856 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:00.723+0000 7f4675635700 1 -- 172.21.15.47:0/1106936080 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f466c00b200 con 0x7f4670070430 2024-01-18T06:48:00.856 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:00.723+0000 7f4675635700 1 --2- 172.21.15.47:0/1106936080 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4670070430 0x7f467010cbd0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f466c00b190 tx=0x7f466c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:00.856 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:00.723+0000 7f4661ffb700 1 -- 172.21.15.47:0/1106936080 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f466c01b410 con 0x7f4670070430 2024-01-18T06:48:00.856 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:00.723+0000 7f4677899700 1 -- 172.21.15.47:0/1106936080 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f467010d2c0 con 0x7f4670070430 2024-01-18T06:48:00.856 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:00.723+0000 7f4661ffb700 1 -- 172.21.15.47:0/1106936080 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f466c01b9e0 con 0x7f4670070430 2024-01-18T06:48:00.856 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:00.723+0000 7f4661ffb700 1 -- 172.21.15.47:0/1106936080 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f466c01a880 con 0x7f4670070430 2024-01-18T06:48:00.856 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:00.723+0000 7f4677899700 1 -- 172.21.15.47:0/1106936080 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f467010d6d0 con 0x7f4670070430 2024-01-18T06:48:00.856 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:00.724+0000 7f4661ffb700 1 -- 172.21.15.47:0/1106936080 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f466c00f070 con 0x7f4670070430 2024-01-18T06:48:00.856 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:00.724+0000 7f4677899700 1 -- 172.21.15.47:0/1106936080 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4670045ad0 con 0x7f4670070430 2024-01-18T06:48:00.856 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:00.724+0000 7f4661ffb700 1 --2- 172.21.15.47:0/1106936080 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f465802fd50 0x7f4658032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:00.857 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:00.724+0000 7f4661ffb700 1 -- 172.21.15.47:0/1106936080 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f466c0412e0 con 0x7f4670070430 2024-01-18T06:48:00.857 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:00.724+0000 7f4674e34700 1 --2- 172.21.15.47:0/1106936080 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f465802fd50 0x7f4658032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:00.857 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:00.725+0000 7f4674e34700 1 --2- 172.21.15.47:0/1106936080 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f465802fd50 0x7f4658032210 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f46640096a0 tx=0x7f4664006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:00.857 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:00.729+0000 7f4661ffb700 1 -- 172.21.15.47:0/1106936080 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f466c010360 con 0x7f4670070430 2024-01-18T06:48:00.890 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:00.889+0000 7f4677899700 1 -- 172.21.15.47:0/1106936080 --> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f467010e010 con 0x7f465802fd50 2024-01-18T06:48:01.316 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:01.316+0000 7f4661ffb700 1 -- 172.21.15.47:0/1106936080 <== mgr.14114 v2:172.21.15.47:6800/3978480908 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f467010e010 con 0x7f465802fd50 2024-01-18T06:48:01.318 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:01.317+0000 7f4677899700 1 -- 172.21.15.47:0/1106936080 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f465802fd50 msgr2=0x7f4658032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:01.319 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:01.317+0000 7f4677899700 1 --2- 172.21.15.47:0/1106936080 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f465802fd50 0x7f4658032210 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f46640096a0 tx=0x7f4664006b40).stop 2024-01-18T06:48:01.319 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:01.317+0000 7f4677899700 1 -- 172.21.15.47:0/1106936080 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4670070430 msgr2=0x7f467010cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:01.319 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:01.317+0000 7f4677899700 1 --2- 172.21.15.47:0/1106936080 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4670070430 0x7f467010cbd0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f466c00b190 tx=0x7f466c006ee0).stop 2024-01-18T06:48:01.319 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:01.317+0000 7f4677899700 1 -- 172.21.15.47:0/1106936080 shutdown_connections 2024-01-18T06:48:01.319 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:01.317+0000 7f4677899700 1 --2- 172.21.15.47:0/1106936080 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f465802fd50 0x7f4658032210 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:01.319 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:01.317+0000 7f4677899700 1 --2- 172.21.15.47:0/1106936080 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4670070430 0x7f467010cbd0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:01.319 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:01.317+0000 7f4677899700 1 -- 172.21.15.47:0/1106936080 >> 172.21.15.47:0/1106936080 conn(0x7f46700f63c0 msgr2=0x7f46700f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:48:01.319 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:01.318+0000 7f4677899700 1 -- 172.21.15.47:0/1106936080 shutdown_connections 2024-01-18T06:48:01.319 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:01.318+0000 7f4677899700 1 -- 172.21.15.47:0/1106936080 wait complete. 2024-01-18T06:48:02.589 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:02 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cephadm 2024-01-18T06:48:00.891969+0000 mgr.smithi047.fuhzzr (mgr.14114) 6 : cephadm [INF] Generating ssh key... 2024-01-18T06:48:02.589 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:02 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:01.315117+0000 mon.smithi047 (mon.0) 52 : audit [INF] from='mgr.14114 172.21.15.47:0/2934400552' entity='mgr.smithi047.fuhzzr' 2024-01-18T06:48:02.589 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:02 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:01.316742+0000 mon.smithi047 (mon.0) 53 : audit [INF] from='mgr.14114 172.21.15.47:0/2934400552' entity='mgr.smithi047.fuhzzr' 2024-01-18T06:48:03.515 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/653e4012-b5cd-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpck5s1jz7:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcg0aieo2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2024-01-18T06:48:04.674 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.577+0000 7f9f53a72700 1 Processor -- start 2024-01-18T06:48:04.674 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.578+0000 7f9f53a72700 1 -- start start 2024-01-18T06:48:04.674 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.578+0000 7f9f53a72700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f4c0fab80 0x7f9f4c0fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:04.674 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.578+0000 7f9f53a72700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4c0fb490 con 0x7f9f4c0fab80 2024-01-18T06:48:04.674 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.578+0000 7f9f5180e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f4c0fab80 0x7f9f4c0fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:04.674 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.578+0000 7f9f5180e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f4c0fab80 0x7f9f4c0fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57682/0 (socket says 172.21.15.47:57682) 2024-01-18T06:48:04.674 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.578+0000 7f9f5180e700 1 -- 172.21.15.47:0/1803809213 learned_addr learned my addr 172.21.15.47:0/1803809213 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:48:04.677 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.578+0000 7f9f5180e700 1 -- 172.21.15.47:0/1803809213 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f4c0fb5d0 con 0x7f9f4c0fab80 2024-01-18T06:48:04.678 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.578+0000 7f9f5180e700 1 --2- 172.21.15.47:0/1803809213 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f4c0fab80 0x7f9f4c0fafa0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f9f4800b5f0 tx=0x7f9f48006ee0).ready entity=mon.0 client_cookie=7174d8d616bdec7b server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:04.678 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.578+0000 7f9f5080c700 1 -- 172.21.15.47:0/1803809213 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f4800d600 con 0x7f9f4c0fab80 2024-01-18T06:48:04.678 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.579+0000 7f9f5080c700 1 -- 172.21.15.47:0/1803809213 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9f4800dbd0 con 0x7f9f4c0fab80 2024-01-18T06:48:04.678 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.579+0000 7f9f5080c700 1 -- 172.21.15.47:0/1803809213 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f48011c00 con 0x7f9f4c0fab80 2024-01-18T06:48:04.678 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.579+0000 7f9f53a72700 1 -- 172.21.15.47:0/1803809213 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f4c0fab80 msgr2=0x7f9f4c0fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:04.678 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.579+0000 7f9f53a72700 1 --2- 172.21.15.47:0/1803809213 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f4c0fab80 0x7f9f4c0fafa0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f9f4800b5f0 tx=0x7f9f48006ee0).stop 2024-01-18T06:48:04.678 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.579+0000 7f9f53a72700 1 -- 172.21.15.47:0/1803809213 shutdown_connections 2024-01-18T06:48:04.678 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.579+0000 7f9f53a72700 1 --2- 172.21.15.47:0/1803809213 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f4c0fab80 0x7f9f4c0fafa0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:04.678 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.579+0000 7f9f53a72700 1 -- 172.21.15.47:0/1803809213 >> 172.21.15.47:0/1803809213 conn(0x7f9f4c0f63c0 msgr2=0x7f9f4c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:48:04.678 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.579+0000 7f9f53a72700 1 -- 172.21.15.47:0/1803809213 shutdown_connections 2024-01-18T06:48:04.678 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.579+0000 7f9f53a72700 1 -- 172.21.15.47:0/1803809213 wait complete. 2024-01-18T06:48:04.678 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.580+0000 7f9f53a72700 1 Processor -- start 2024-01-18T06:48:04.678 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.580+0000 7f9f53a72700 1 -- start start 2024-01-18T06:48:04.678 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.580+0000 7f9f53a72700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f4c0fab80 0x7f9f4c10d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:04.678 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.580+0000 7f9f53a72700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4c070450 con 0x7f9f4c0fab80 2024-01-18T06:48:04.679 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.580+0000 7f9f5180e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f4c0fab80 0x7f9f4c10d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:04.679 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.580+0000 7f9f5180e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f4c0fab80 0x7f9f4c10d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57690/0 (socket says 172.21.15.47:57690) 2024-01-18T06:48:04.679 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.580+0000 7f9f5180e700 1 -- 172.21.15.47:0/2267793577 learned_addr learned my addr 172.21.15.47:0/2267793577 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:48:04.679 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.580+0000 7f9f5180e700 1 -- 172.21.15.47:0/2267793577 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f4800b200 con 0x7f9f4c0fab80 2024-01-18T06:48:04.679 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.581+0000 7f9f5180e700 1 --2- 172.21.15.47:0/2267793577 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f4c0fab80 0x7f9f4c10d720 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f9f4800b1d0 tx=0x7f9f48006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:04.679 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.581+0000 7f9f3a7fc700 1 -- 172.21.15.47:0/2267793577 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f4801b410 con 0x7f9f4c0fab80 2024-01-18T06:48:04.679 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.581+0000 7f9f3a7fc700 1 -- 172.21.15.47:0/2267793577 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9f4801b9e0 con 0x7f9f4c0fab80 2024-01-18T06:48:04.679 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.581+0000 7f9f53a72700 1 -- 172.21.15.47:0/2267793577 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f4c070650 con 0x7f9f4c0fab80 2024-01-18T06:48:04.679 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.581+0000 7f9f3a7fc700 1 -- 172.21.15.47:0/2267793577 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f4801a840 con 0x7f9f4c0fab80 2024-01-18T06:48:04.679 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.581+0000 7f9f53a72700 1 -- 172.21.15.47:0/2267793577 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f4c10ddd0 con 0x7f9f4c0fab80 2024-01-18T06:48:04.679 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.582+0000 7f9f3a7fc700 1 -- 172.21.15.47:0/2267793577 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f9f4800f070 con 0x7f9f4c0fab80 2024-01-18T06:48:04.679 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.582+0000 7f9f3a7fc700 1 --2- 172.21.15.47:0/2267793577 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f9f3402fdb0 0x7f9f34032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:04.679 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.582+0000 7f9f53a72700 1 -- 172.21.15.47:0/2267793577 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f4c045ad0 con 0x7f9f4c0fab80 2024-01-18T06:48:04.680 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.582+0000 7f9f3a7fc700 1 -- 172.21.15.47:0/2267793577 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f9f480411a0 con 0x7f9f4c0fab80 2024-01-18T06:48:04.680 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.582+0000 7f9f5100d700 1 --2- 172.21.15.47:0/2267793577 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f9f3402fdb0 0x7f9f34032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:04.680 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.582+0000 7f9f5100d700 1 --2- 172.21.15.47:0/2267793577 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f9f3402fdb0 0x7f9f34032270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f9f3c0096a0 tx=0x7f9f3c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:04.680 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.586+0000 7f9f3a7fc700 1 -- 172.21.15.47:0/2267793577 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9f48016070 con 0x7f9f4c0fab80 2024-01-18T06:48:04.751 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.751+0000 7f9f53a72700 1 -- 172.21.15.47:0/2267793577 --> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f9f4c057f10 con 0x7f9f3402fdb0 2024-01-18T06:48:04.752 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.751+0000 7f9f3a7fc700 1 -- 172.21.15.47:0/2267793577 <== mgr.14114 v2:172.21.15.47:6800/3978480908 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f9f4c057f10 con 0x7f9f3402fdb0 2024-01-18T06:48:04.752 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTUlFvL9QJzLbw5AICxcbAnlSDLfp3OyWF2qfKGQW5xqA24NQD/ytwqGXzFdU7CQB6S/43aYjhvz1aoYDblZCYmkQ0zmszLQRF3qzQ88zfyWrZrG6Yjl5L0sA1S8zQkd9fSfXXN0C5Zx5NYa3bfS59JmCoRWYX6Hm0t/IokIGAwQBu+ti0GTCrJvnkrMSRoAjny3OtFQ3DSsrOgzsLBOM8Ij3xky25fhFunkjzMLBlPTT3IogH2j5fwlr9XCk4ro8k+N735KuMN1JS8PJiNVpJWV+CYVybH1gMGK0hPelKMzARPJu4fi6nFeR7jQ3DxXzqq5O+qPWmH8JfMDSHXM+eMbKommvet1WW2b5GsmkTx5TplWn1LrplbMb1NPSzJC/7Zz04Tg9Bg3Fc38xk6vLXTMQYYXzCTOD8MY26IX07qq5KKDNCImtxdR0+Bnor7WSTJK5yYGYUG9Jy29mAXrRUoKdZAc1KxngjQkejKQCyx4YxplK6RoKGKmV3KnZ+des= ceph-653e4012-b5cd-11ee-95ae-87774f69a715 2024-01-18T06:48:04.754 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.753+0000 7f9f53a72700 1 -- 172.21.15.47:0/2267793577 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f9f3402fdb0 msgr2=0x7f9f34032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:04.754 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.753+0000 7f9f53a72700 1 --2- 172.21.15.47:0/2267793577 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f9f3402fdb0 0x7f9f34032270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f9f3c0096a0 tx=0x7f9f3c006b40).stop 2024-01-18T06:48:04.754 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.753+0000 7f9f53a72700 1 -- 172.21.15.47:0/2267793577 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f4c0fab80 msgr2=0x7f9f4c10d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:04.754 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.753+0000 7f9f53a72700 1 --2- 172.21.15.47:0/2267793577 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f4c0fab80 0x7f9f4c10d720 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f9f4800b1d0 tx=0x7f9f48006ee0).stop 2024-01-18T06:48:04.754 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.753+0000 7f9f53a72700 1 -- 172.21.15.47:0/2267793577 shutdown_connections 2024-01-18T06:48:04.754 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.753+0000 7f9f53a72700 1 --2- 172.21.15.47:0/2267793577 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f9f3402fdb0 0x7f9f34032270 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:04.754 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.753+0000 7f9f53a72700 1 --2- 172.21.15.47:0/2267793577 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f4c0fab80 0x7f9f4c10d720 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:04.755 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.753+0000 7f9f53a72700 1 -- 172.21.15.47:0/2267793577 >> 172.21.15.47:0/2267793577 conn(0x7f9f4c0f63c0 msgr2=0x7f9f4c0f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:48:04.755 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.753+0000 7f9f53a72700 1 -- 172.21.15.47:0/2267793577 shutdown_connections 2024-01-18T06:48:04.755 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:04.753+0000 7f9f53a72700 1 -- 172.21.15.47:0/2267793577 wait complete. 2024-01-18T06:48:06.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:05 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:04.753180+0000 mgr.smithi047.fuhzzr (mgr.14114) 7 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-01-18T06:48:06.632 INFO:teuthology.orchestra.run.smithi047.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-01-18T06:48:06.632 INFO:teuthology.orchestra.run.smithi047.stderr:Adding key to root@localhost authorized_keys... 2024-01-18T06:48:06.633 INFO:teuthology.orchestra.run.smithi047.stderr:Adding host smithi047... 2024-01-18T06:48:06.633 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/653e4012-b5cd-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpck5s1jz7:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcg0aieo2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi047 172.21.15.47 2024-01-18T06:48:07.090 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.087+0000 7f117cf94700 1 Processor -- start 2024-01-18T06:48:07.090 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.087+0000 7f117cf94700 1 -- start start 2024-01-18T06:48:07.090 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.087+0000 7f117cf94700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1178070430 0x7f11780fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:07.090 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.087+0000 7f117cf94700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11780fedd0 con 0x7f1178070430 2024-01-18T06:48:07.090 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.088+0000 7f117659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1178070430 0x7f11780fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:07.091 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.088+0000 7f117659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1178070430 0x7f11780fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57694/0 (socket says 172.21.15.47:57694) 2024-01-18T06:48:07.091 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.088+0000 7f117659c700 1 -- 172.21.15.47:0/1175942709 learned_addr learned my addr 172.21.15.47:0/1175942709 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:48:07.091 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.088+0000 7f117659c700 1 -- 172.21.15.47:0/1175942709 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11780fef10 con 0x7f1178070430 2024-01-18T06:48:07.091 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.088+0000 7f117659c700 1 --2- 172.21.15.47:0/1175942709 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1178070430 0x7f11780fe8e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f116c00b5f0 tx=0x7f116c006ee0).ready entity=mon.0 client_cookie=6bc0f312e22ab344 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:07.091 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.088+0000 7f117559a700 1 -- 172.21.15.47:0/1175942709 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f116c00d600 con 0x7f1178070430 2024-01-18T06:48:07.091 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.088+0000 7f117559a700 1 -- 172.21.15.47:0/1175942709 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f116c00dbd0 con 0x7f1178070430 2024-01-18T06:48:07.091 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.088+0000 7f117559a700 1 -- 172.21.15.47:0/1175942709 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f116c011c00 con 0x7f1178070430 2024-01-18T06:48:07.091 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.089+0000 7f117cf94700 1 -- 172.21.15.47:0/1175942709 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1178070430 msgr2=0x7f11780fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:07.091 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.089+0000 7f117cf94700 1 --2- 172.21.15.47:0/1175942709 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1178070430 0x7f11780fe8e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f116c00b5f0 tx=0x7f116c006ee0).stop 2024-01-18T06:48:07.091 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.089+0000 7f117cf94700 1 -- 172.21.15.47:0/1175942709 shutdown_connections 2024-01-18T06:48:07.091 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.089+0000 7f117cf94700 1 --2- 172.21.15.47:0/1175942709 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1178070430 0x7f11780fe8e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:07.091 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.089+0000 7f117cf94700 1 -- 172.21.15.47:0/1175942709 >> 172.21.15.47:0/1175942709 conn(0x7f11780f63c0 msgr2=0x7f11780f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:48:07.091 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.089+0000 7f117cf94700 1 -- 172.21.15.47:0/1175942709 shutdown_connections 2024-01-18T06:48:07.091 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.089+0000 7f117cf94700 1 -- 172.21.15.47:0/1175942709 wait complete. 2024-01-18T06:48:07.092 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.090+0000 7f117cf94700 1 Processor -- start 2024-01-18T06:48:07.092 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.090+0000 7f117cf94700 1 -- start start 2024-01-18T06:48:07.094 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.090+0000 7f117cf94700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1178070430 0x7f117810aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:07.094 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.090+0000 7f117cf94700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117810aef0 con 0x7f1178070430 2024-01-18T06:48:07.094 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.090+0000 7f117659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1178070430 0x7f117810aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:07.094 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.090+0000 7f117659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1178070430 0x7f117810aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57702/0 (socket says 172.21.15.47:57702) 2024-01-18T06:48:07.094 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.090+0000 7f117659c700 1 -- 172.21.15.47:0/4162619371 learned_addr learned my addr 172.21.15.47:0/4162619371 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:48:07.094 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.090+0000 7f117659c700 1 -- 172.21.15.47:0/4162619371 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f116c00b200 con 0x7f1178070430 2024-01-18T06:48:07.094 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.090+0000 7f117659c700 1 --2- 172.21.15.47:0/4162619371 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1178070430 0x7f117810aa00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f116c00b190 tx=0x7f116c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:07.094 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.091+0000 7f1166ffd700 1 -- 172.21.15.47:0/4162619371 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f116c01b410 con 0x7f1178070430 2024-01-18T06:48:07.094 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.091+0000 7f1166ffd700 1 -- 172.21.15.47:0/4162619371 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f116c01b9e0 con 0x7f1178070430 2024-01-18T06:48:07.094 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.091+0000 7f117cf94700 1 -- 172.21.15.47:0/4162619371 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f117810b0f0 con 0x7f1178070430 2024-01-18T06:48:07.094 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.091+0000 7f1166ffd700 1 -- 172.21.15.47:0/4162619371 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f116c01a880 con 0x7f1178070430 2024-01-18T06:48:07.094 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.091+0000 7f117cf94700 1 -- 172.21.15.47:0/4162619371 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f117810b500 con 0x7f1178070430 2024-01-18T06:48:07.095 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.092+0000 7f1166ffd700 1 -- 172.21.15.47:0/4162619371 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f116c00f070 con 0x7f1178070430 2024-01-18T06:48:07.095 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.092+0000 7f1166ffd700 1 --2- 172.21.15.47:0/4162619371 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f115c02fd50 0x7f115c032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:07.095 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.092+0000 7f1166ffd700 1 -- 172.21.15.47:0/4162619371 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f116c0412a0 con 0x7f1178070430 2024-01-18T06:48:07.095 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.092+0000 7f1175d9b700 1 --2- 172.21.15.47:0/4162619371 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f115c02fd50 0x7f115c032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:07.095 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.092+0000 7f117cf94700 1 -- 172.21.15.47:0/4162619371 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1178059100 con 0x7f1178070430 2024-01-18T06:48:07.095 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.092+0000 7f1175d9b700 1 --2- 172.21.15.47:0/4162619371 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f115c02fd50 0x7f115c032210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f11680096a0 tx=0x7f1168006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:07.096 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.096+0000 7f1166ffd700 1 -- 172.21.15.47:0/4162619371 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f116c016070 con 0x7f1178070430 2024-01-18T06:48:07.277 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.276+0000 7f117cf94700 1 -- 172.21.15.47:0/4162619371 --> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi047", "addr": "172.21.15.47", "target": ["mon-mgr", ""]}) v1 -- 0x7f11781048f0 con 0x7f115c02fd50 2024-01-18T06:48:07.880 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: Added host 'smithi047' 2024-01-18T06:48:07.880 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.878+0000 7f1166ffd700 1 -- 172.21.15.47:0/4162619371 <== mgr.14114 v2:172.21.15.47:6800/3978480908 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f11781048f0 con 0x7f115c02fd50 2024-01-18T06:48:07.881 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.881+0000 7f117cf94700 1 -- 172.21.15.47:0/4162619371 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f115c02fd50 msgr2=0x7f115c032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:07.882 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.881+0000 7f117cf94700 1 --2- 172.21.15.47:0/4162619371 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f115c02fd50 0x7f115c032210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f11680096a0 tx=0x7f1168006b40).stop 2024-01-18T06:48:07.882 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.881+0000 7f117cf94700 1 -- 172.21.15.47:0/4162619371 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1178070430 msgr2=0x7f117810aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:07.882 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.881+0000 7f117cf94700 1 --2- 172.21.15.47:0/4162619371 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1178070430 0x7f117810aa00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f116c00b190 tx=0x7f116c006ee0).stop 2024-01-18T06:48:07.882 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.881+0000 7f117cf94700 1 -- 172.21.15.47:0/4162619371 shutdown_connections 2024-01-18T06:48:07.883 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.881+0000 7f117cf94700 1 --2- 172.21.15.47:0/4162619371 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f115c02fd50 0x7f115c032210 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:07.883 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.881+0000 7f117cf94700 1 --2- 172.21.15.47:0/4162619371 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1178070430 0x7f117810aa00 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:07.883 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.881+0000 7f117cf94700 1 -- 172.21.15.47:0/4162619371 >> 172.21.15.47:0/4162619371 conn(0x7f11780f63c0 msgr2=0x7f11780f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:48:07.883 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.881+0000 7f117cf94700 1 -- 172.21.15.47:0/4162619371 shutdown_connections 2024-01-18T06:48:07.883 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:07.881+0000 7f117cf94700 1 -- 172.21.15.47:0/4162619371 wait complete. 2024-01-18T06:48:08.047 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:07 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:07.279012+0000 mgr.smithi047.fuhzzr (mgr.14114) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi047", "addr": "172.21.15.47", "target": ["mon-mgr", ""]}]: dispatch 2024-01-18T06:48:08.206 INFO:teuthology.orchestra.run.smithi047.stderr:Deploying mon service with default placement... 2024-01-18T06:48:08.206 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/653e4012-b5cd-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpck5s1jz7:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcg0aieo2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2024-01-18T06:48:08.585 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.584+0000 7f4b7d57a700 1 Processor -- start 2024-01-18T06:48:08.585 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.584+0000 7f4b7d57a700 1 -- start start 2024-01-18T06:48:08.585 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.584+0000 7f4b7d57a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b780703f0 0x7f4b780fe7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:08.585 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.584+0000 7f4b7d57a700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b780fec90 con 0x7f4b780703f0 2024-01-18T06:48:08.585 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.584+0000 7f4b76ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b780703f0 0x7f4b780fe7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:08.586 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.584+0000 7f4b76ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b780703f0 0x7f4b780fe7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57718/0 (socket says 172.21.15.47:57718) 2024-01-18T06:48:08.586 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.584+0000 7f4b76ffd700 1 -- 172.21.15.47:0/2397767608 learned_addr learned my addr 172.21.15.47:0/2397767608 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:48:08.586 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.585+0000 7f4b76ffd700 1 -- 172.21.15.47:0/2397767608 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b780fedd0 con 0x7f4b780703f0 2024-01-18T06:48:08.586 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.585+0000 7f4b76ffd700 1 --2- 172.21.15.47:0/2397767608 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b780703f0 0x7f4b780fe7a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f4b6800b5f0 tx=0x7f4b68006ee0).ready entity=mon.0 client_cookie=8c65fd87c639fb39 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:08.588 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.585+0000 7f4b75ffb700 1 -- 172.21.15.47:0/2397767608 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b6800d600 con 0x7f4b780703f0 2024-01-18T06:48:08.588 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.585+0000 7f4b75ffb700 1 -- 172.21.15.47:0/2397767608 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4b6800dbd0 con 0x7f4b780703f0 2024-01-18T06:48:08.588 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.585+0000 7f4b75ffb700 1 -- 172.21.15.47:0/2397767608 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b68011c00 con 0x7f4b780703f0 2024-01-18T06:48:08.588 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.585+0000 7f4b7d57a700 1 -- 172.21.15.47:0/2397767608 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b780703f0 msgr2=0x7f4b780fe7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:08.588 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.585+0000 7f4b7d57a700 1 --2- 172.21.15.47:0/2397767608 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b780703f0 0x7f4b780fe7a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f4b6800b5f0 tx=0x7f4b68006ee0).stop 2024-01-18T06:48:08.588 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.586+0000 7f4b7d57a700 1 -- 172.21.15.47:0/2397767608 shutdown_connections 2024-01-18T06:48:08.589 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.586+0000 7f4b7d57a700 1 --2- 172.21.15.47:0/2397767608 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b780703f0 0x7f4b780fe7a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:08.589 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.586+0000 7f4b7d57a700 1 -- 172.21.15.47:0/2397767608 >> 172.21.15.47:0/2397767608 conn(0x7f4b780f6240 msgr2=0x7f4b780f8660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:48:08.589 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.586+0000 7f4b7d57a700 1 -- 172.21.15.47:0/2397767608 shutdown_connections 2024-01-18T06:48:08.589 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.586+0000 7f4b7d57a700 1 -- 172.21.15.47:0/2397767608 wait complete. 2024-01-18T06:48:08.589 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.586+0000 7f4b7d57a700 1 Processor -- start 2024-01-18T06:48:08.589 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.586+0000 7f4b7d57a700 1 -- start start 2024-01-18T06:48:08.589 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.586+0000 7f4b7d57a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b780703f0 0x7f4b7810a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:08.589 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.587+0000 7f4b7d57a700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7810ad30 con 0x7f4b780703f0 2024-01-18T06:48:08.589 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.587+0000 7f4b76ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b780703f0 0x7f4b7810a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:08.589 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.587+0000 7f4b76ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b780703f0 0x7f4b7810a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57726/0 (socket says 172.21.15.47:57726) 2024-01-18T06:48:08.589 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.587+0000 7f4b76ffd700 1 -- 172.21.15.47:0/588285274 learned_addr learned my addr 172.21.15.47:0/588285274 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:48:08.589 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.587+0000 7f4b76ffd700 1 -- 172.21.15.47:0/588285274 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b6800b200 con 0x7f4b780703f0 2024-01-18T06:48:08.589 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.587+0000 7f4b76ffd700 1 --2- 172.21.15.47:0/588285274 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b780703f0 0x7f4b7810a840 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f4b6800b190 tx=0x7f4b68006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:08.589 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.587+0000 7f4b5f7fe700 1 -- 172.21.15.47:0/588285274 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b6801b410 con 0x7f4b780703f0 2024-01-18T06:48:08.589 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.587+0000 7f4b5f7fe700 1 -- 172.21.15.47:0/588285274 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4b6801b9e0 con 0x7f4b780703f0 2024-01-18T06:48:08.590 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.587+0000 7f4b7d57a700 1 -- 172.21.15.47:0/588285274 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b7810af30 con 0x7f4b780703f0 2024-01-18T06:48:08.590 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.587+0000 7f4b5f7fe700 1 -- 172.21.15.47:0/588285274 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b6801a880 con 0x7f4b780703f0 2024-01-18T06:48:08.590 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.587+0000 7f4b7d57a700 1 -- 172.21.15.47:0/588285274 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b7810b340 con 0x7f4b780703f0 2024-01-18T06:48:08.590 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.588+0000 7f4b5f7fe700 1 -- 172.21.15.47:0/588285274 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f4b6800f070 con 0x7f4b780703f0 2024-01-18T06:48:08.591 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.588+0000 7f4b5f7fe700 1 --2- 172.21.15.47:0/588285274 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f4b6402fdb0 0x7f4b64032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:08.591 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.588+0000 7f4b5f7fe700 1 -- 172.21.15.47:0/588285274 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f4b680411e0 con 0x7f4b780703f0 2024-01-18T06:48:08.591 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.588+0000 7f4b7d57a700 1 -- 172.21.15.47:0/588285274 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b78045ad0 con 0x7f4b780703f0 2024-01-18T06:48:08.591 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.588+0000 7f4b767fc700 1 --2- 172.21.15.47:0/588285274 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f4b6402fdb0 0x7f4b64032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:08.591 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.589+0000 7f4b767fc700 1 --2- 172.21.15.47:0/588285274 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f4b6402fdb0 0x7f4b64032270 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f4b6c0096a0 tx=0x7f4b6c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:08.593 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.593+0000 7f4b5f7fe700 1 -- 172.21.15.47:0/588285274 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4b68025460 con 0x7f4b780703f0 2024-01-18T06:48:08.778 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.778+0000 7f4b7d57a700 1 -- 172.21.15.47:0/588285274 --> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f4b781046e0 con 0x7f4b6402fdb0 2024-01-18T06:48:08.782 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.782+0000 7f4b5f7fe700 1 -- 172.21.15.47:0/588285274 <== mgr.14114 v2:172.21.15.47:6800/3978480908 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f4b781046e0 con 0x7f4b6402fdb0 2024-01-18T06:48:08.782 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: Scheduled mon update... 2024-01-18T06:48:08.784 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.783+0000 7f4b7d57a700 1 -- 172.21.15.47:0/588285274 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f4b6402fdb0 msgr2=0x7f4b64032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:08.785 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.783+0000 7f4b7d57a700 1 --2- 172.21.15.47:0/588285274 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f4b6402fdb0 0x7f4b64032270 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f4b6c0096a0 tx=0x7f4b6c006b40).stop 2024-01-18T06:48:08.785 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.783+0000 7f4b7d57a700 1 -- 172.21.15.47:0/588285274 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b780703f0 msgr2=0x7f4b7810a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:08.785 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.783+0000 7f4b7d57a700 1 --2- 172.21.15.47:0/588285274 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b780703f0 0x7f4b7810a840 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f4b6800b190 tx=0x7f4b68006ee0).stop 2024-01-18T06:48:08.785 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.783+0000 7f4b7d57a700 1 -- 172.21.15.47:0/588285274 shutdown_connections 2024-01-18T06:48:08.785 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.783+0000 7f4b7d57a700 1 --2- 172.21.15.47:0/588285274 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f4b6402fdb0 0x7f4b64032270 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:08.785 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.783+0000 7f4b7d57a700 1 --2- 172.21.15.47:0/588285274 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4b780703f0 0x7f4b7810a840 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:08.785 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.783+0000 7f4b7d57a700 1 -- 172.21.15.47:0/588285274 >> 172.21.15.47:0/588285274 conn(0x7f4b780f6240 msgr2=0x7f4b780f7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:48:08.785 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.783+0000 7f4b7d57a700 1 -- 172.21.15.47:0/588285274 shutdown_connections 2024-01-18T06:48:08.785 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:08.783+0000 7f4b7d57a700 1 -- 172.21.15.47:0/588285274 wait complete. 2024-01-18T06:48:09.116 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:08 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:07.879446+0000 mon.smithi047 (mon.0) 54 : audit [INF] from='mgr.14114 172.21.15.47:0/2934400552' entity='mgr.smithi047.fuhzzr' 2024-01-18T06:48:09.117 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:08 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cephadm 2024-01-18T06:48:07.879755+0000 mgr.smithi047.fuhzzr (mgr.14114) 9 : cephadm [INF] Added host smithi047 2024-01-18T06:48:09.117 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:08 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:07.880017+0000 mon.smithi047 (mon.0) 55 : audit [DBG] from='mgr.14114 172.21.15.47:0/2934400552' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-18T06:48:09.117 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:08 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:08.783236+0000 mon.smithi047 (mon.0) 56 : audit [INF] from='mgr.14114 172.21.15.47:0/2934400552' entity='mgr.smithi047.fuhzzr' 2024-01-18T06:48:09.147 INFO:teuthology.orchestra.run.smithi047.stderr:Deploying mgr service with default placement... 2024-01-18T06:48:09.147 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/653e4012-b5cd-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpck5s1jz7:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcg0aieo2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2024-01-18T06:48:09.499 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.494+0000 7fccb300d700 1 Processor -- start 2024-01-18T06:48:09.499 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.494+0000 7fccb300d700 1 -- start start 2024-01-18T06:48:09.499 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.494+0000 7fccb300d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fccac070430 0x7fccac0fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:09.499 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.494+0000 7fccb300d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccac0fedd0 con 0x7fccac070430 2024-01-18T06:48:09.499 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.495+0000 7fccb0da9700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fccac070430 0x7fccac0fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:09.499 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.495+0000 7fccb0da9700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fccac070430 0x7fccac0fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57738/0 (socket says 172.21.15.47:57738) 2024-01-18T06:48:09.499 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.495+0000 7fccb0da9700 1 -- 172.21.15.47:0/119976494 learned_addr learned my addr 172.21.15.47:0/119976494 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:48:09.499 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.495+0000 7fccb0da9700 1 -- 172.21.15.47:0/119976494 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccac0fef10 con 0x7fccac070430 2024-01-18T06:48:09.499 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.495+0000 7fccb0da9700 1 --2- 172.21.15.47:0/119976494 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fccac070430 0x7fccac0fe8e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fcca000b5f0 tx=0x7fcca0006ee0).ready entity=mon.0 client_cookie=9af9957c03a6145a server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:09.499 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.495+0000 7fccab7fe700 1 -- 172.21.15.47:0/119976494 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcca000d600 con 0x7fccac070430 2024-01-18T06:48:09.499 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.495+0000 7fccab7fe700 1 -- 172.21.15.47:0/119976494 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fcca000dbd0 con 0x7fccac070430 2024-01-18T06:48:09.500 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.495+0000 7fccab7fe700 1 -- 172.21.15.47:0/119976494 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcca0011c00 con 0x7fccac070430 2024-01-18T06:48:09.500 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.496+0000 7fccb300d700 1 -- 172.21.15.47:0/119976494 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fccac070430 msgr2=0x7fccac0fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:09.500 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.496+0000 7fccb300d700 1 --2- 172.21.15.47:0/119976494 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fccac070430 0x7fccac0fe8e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fcca000b5f0 tx=0x7fcca0006ee0).stop 2024-01-18T06:48:09.500 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.496+0000 7fccb300d700 1 -- 172.21.15.47:0/119976494 shutdown_connections 2024-01-18T06:48:09.500 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.496+0000 7fccb300d700 1 --2- 172.21.15.47:0/119976494 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fccac070430 0x7fccac0fe8e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:09.500 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.496+0000 7fccb300d700 1 -- 172.21.15.47:0/119976494 >> 172.21.15.47:0/119976494 conn(0x7fccac0f63c0 msgr2=0x7fccac0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:48:09.500 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.496+0000 7fccb300d700 1 -- 172.21.15.47:0/119976494 shutdown_connections 2024-01-18T06:48:09.500 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.496+0000 7fccb300d700 1 -- 172.21.15.47:0/119976494 wait complete. 2024-01-18T06:48:09.500 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.497+0000 7fccb300d700 1 Processor -- start 2024-01-18T06:48:09.500 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.497+0000 7fccb300d700 1 -- start start 2024-01-18T06:48:09.500 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.497+0000 7fccb300d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fccac070430 0x7fccac0fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:09.500 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.497+0000 7fccb300d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccac0fa7e0 con 0x7fccac070430 2024-01-18T06:48:09.500 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.497+0000 7fccb0da9700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fccac070430 0x7fccac0fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:09.501 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.497+0000 7fccb0da9700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fccac070430 0x7fccac0fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57748/0 (socket says 172.21.15.47:57748) 2024-01-18T06:48:09.501 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.497+0000 7fccb0da9700 1 -- 172.21.15.47:0/423576602 learned_addr learned my addr 172.21.15.47:0/423576602 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:48:09.501 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.498+0000 7fccb0da9700 1 -- 172.21.15.47:0/423576602 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca000b200 con 0x7fccac070430 2024-01-18T06:48:09.501 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.498+0000 7fccb0da9700 1 --2- 172.21.15.47:0/423576602 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fccac070430 0x7fccac0fc360 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fcca000b190 tx=0x7fcca0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:09.501 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.498+0000 7fcca97fa700 1 -- 172.21.15.47:0/423576602 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcca001b410 con 0x7fccac070430 2024-01-18T06:48:09.501 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.498+0000 7fccb300d700 1 -- 172.21.15.47:0/423576602 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccac0fa980 con 0x7fccac070430 2024-01-18T06:48:09.501 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.498+0000 7fcca97fa700 1 -- 172.21.15.47:0/423576602 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fcca001b9e0 con 0x7fccac070430 2024-01-18T06:48:09.501 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.498+0000 7fcca97fa700 1 -- 172.21.15.47:0/423576602 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcca001a880 con 0x7fccac070430 2024-01-18T06:48:09.501 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.498+0000 7fccb300d700 1 -- 172.21.15.47:0/423576602 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccac0fad90 con 0x7fccac070430 2024-01-18T06:48:09.502 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.499+0000 7fcca97fa700 1 -- 172.21.15.47:0/423576602 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fcca000f070 con 0x7fccac070430 2024-01-18T06:48:09.502 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.499+0000 7fcca97fa700 1 --2- 172.21.15.47:0/423576602 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7fcc9002fd50 0x7fcc90032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:09.502 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.499+0000 7fcca97fa700 1 -- 172.21.15.47:0/423576602 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fcca0046580 con 0x7fccac070430 2024-01-18T06:48:09.502 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.499+0000 7fccb300d700 1 -- 172.21.15.47:0/423576602 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccac045ad0 con 0x7fccac070430 2024-01-18T06:48:09.502 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.499+0000 7fccabfff700 1 --2- 172.21.15.47:0/423576602 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7fcc9002fd50 0x7fcc90032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:09.502 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.500+0000 7fccabfff700 1 --2- 172.21.15.47:0/423576602 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7fcc9002fd50 0x7fcc90032210 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fcc980096a0 tx=0x7fcc98006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:09.504 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.504+0000 7fcca97fa700 1 -- 172.21.15.47:0/423576602 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcca0047060 con 0x7fccac070430 2024-01-18T06:48:09.688 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.688+0000 7fccb300d700 1 -- 172.21.15.47:0/423576602 --> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fccac104900 con 0x7fcc9002fd50 2024-01-18T06:48:09.692 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.692+0000 7fcca97fa700 1 -- 172.21.15.47:0/423576602 <== mgr.14114 v2:172.21.15.47:6800/3978480908 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fccac104900 con 0x7fcc9002fd50 2024-01-18T06:48:09.692 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-01-18T06:48:09.694 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.693+0000 7fccb300d700 1 -- 172.21.15.47:0/423576602 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7fcc9002fd50 msgr2=0x7fcc90032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:09.695 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.693+0000 7fccb300d700 1 --2- 172.21.15.47:0/423576602 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7fcc9002fd50 0x7fcc90032210 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fcc980096a0 tx=0x7fcc98006b40).stop 2024-01-18T06:48:09.695 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.693+0000 7fccb300d700 1 -- 172.21.15.47:0/423576602 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fccac070430 msgr2=0x7fccac0fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:09.695 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.693+0000 7fccb300d700 1 --2- 172.21.15.47:0/423576602 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fccac070430 0x7fccac0fc360 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fcca000b190 tx=0x7fcca0006ee0).stop 2024-01-18T06:48:09.695 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.693+0000 7fccb300d700 1 -- 172.21.15.47:0/423576602 shutdown_connections 2024-01-18T06:48:09.695 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.693+0000 7fccb300d700 1 --2- 172.21.15.47:0/423576602 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7fcc9002fd50 0x7fcc90032210 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:09.695 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.694+0000 7fccb300d700 1 --2- 172.21.15.47:0/423576602 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fccac070430 0x7fccac0fc360 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:09.695 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.694+0000 7fccb300d700 1 -- 172.21.15.47:0/423576602 >> 172.21.15.47:0/423576602 conn(0x7fccac0f63c0 msgr2=0x7fccac0f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:48:09.695 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.694+0000 7fccb300d700 1 -- 172.21.15.47:0/423576602 shutdown_connections 2024-01-18T06:48:09.695 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:09.694+0000 7fccb300d700 1 -- 172.21.15.47:0/423576602 wait complete. 2024-01-18T06:48:10.039 INFO:teuthology.orchestra.run.smithi047.stderr:Deploying crash service with default placement... 2024-01-18T06:48:10.039 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/653e4012-b5cd-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpck5s1jz7:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcg0aieo2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2024-01-18T06:48:10.414 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.413+0000 7f42eb175700 1 Processor -- start 2024-01-18T06:48:10.414 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.413+0000 7f42eb175700 1 -- start start 2024-01-18T06:48:10.414 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.413+0000 7f42eb175700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f42e40fbaf0 0x7f42e40fbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:10.414 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.413+0000 7f42eb175700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e40fc400 con 0x7f42e40fbaf0 2024-01-18T06:48:10.414 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.413+0000 7f42ea173700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f42e40fbaf0 0x7f42e40fbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:10.415 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.413+0000 7f42ea173700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f42e40fbaf0 0x7f42e40fbf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57756/0 (socket says 172.21.15.47:57756) 2024-01-18T06:48:10.415 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.413+0000 7f42ea173700 1 -- 172.21.15.47:0/2045749236 learned_addr learned my addr 172.21.15.47:0/2045749236 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:48:10.418 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.414+0000 7f42ea173700 1 -- 172.21.15.47:0/2045749236 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e40fc540 con 0x7f42e40fbaf0 2024-01-18T06:48:10.418 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.414+0000 7f42ea173700 1 --2- 172.21.15.47:0/2045749236 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f42e40fbaf0 0x7f42e40fbf10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f42e000b5f0 tx=0x7f42e0006ee0).ready entity=mon.0 client_cookie=688e2b4b7941d9d1 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:10.418 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.414+0000 7f42e9171700 1 -- 172.21.15.47:0/2045749236 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42e000d600 con 0x7f42e40fbaf0 2024-01-18T06:48:10.418 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.414+0000 7f42e9171700 1 -- 172.21.15.47:0/2045749236 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f42e000dbd0 con 0x7f42e40fbaf0 2024-01-18T06:48:10.418 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.414+0000 7f42e9171700 1 -- 172.21.15.47:0/2045749236 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42e0011c00 con 0x7f42e40fbaf0 2024-01-18T06:48:10.418 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.414+0000 7f42eb175700 1 -- 172.21.15.47:0/2045749236 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f42e40fbaf0 msgr2=0x7f42e40fbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:10.419 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.414+0000 7f42eb175700 1 --2- 172.21.15.47:0/2045749236 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f42e40fbaf0 0x7f42e40fbf10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f42e000b5f0 tx=0x7f42e0006ee0).stop 2024-01-18T06:48:10.419 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.415+0000 7f42eb175700 1 -- 172.21.15.47:0/2045749236 shutdown_connections 2024-01-18T06:48:10.419 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.415+0000 7f42eb175700 1 --2- 172.21.15.47:0/2045749236 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f42e40fbaf0 0x7f42e40fbf10 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:10.419 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.415+0000 7f42eb175700 1 -- 172.21.15.47:0/2045749236 >> 172.21.15.47:0/2045749236 conn(0x7f42e40f7350 msgr2=0x7f42e40f9770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:48:10.419 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.415+0000 7f42eb175700 1 -- 172.21.15.47:0/2045749236 shutdown_connections 2024-01-18T06:48:10.419 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.415+0000 7f42eb175700 1 -- 172.21.15.47:0/2045749236 wait complete. 2024-01-18T06:48:10.419 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.415+0000 7f42eb175700 1 Processor -- start 2024-01-18T06:48:10.419 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.415+0000 7f42eb175700 1 -- start start 2024-01-18T06:48:10.419 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.415+0000 7f42eb175700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f42e40fbaf0 0x7f42e40725e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:10.419 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.415+0000 7f42eb175700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e4070a60 con 0x7f42e40fbaf0 2024-01-18T06:48:10.419 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.416+0000 7f42ea173700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f42e40fbaf0 0x7f42e40725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:10.419 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.416+0000 7f42ea173700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f42e40fbaf0 0x7f42e40725e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57772/0 (socket says 172.21.15.47:57772) 2024-01-18T06:48:10.419 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.416+0000 7f42ea173700 1 -- 172.21.15.47:0/88768799 learned_addr learned my addr 172.21.15.47:0/88768799 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:48:10.419 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.416+0000 7f42ea173700 1 -- 172.21.15.47:0/88768799 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e000b200 con 0x7f42e40fbaf0 2024-01-18T06:48:10.420 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.416+0000 7f42ea173700 1 --2- 172.21.15.47:0/88768799 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f42e40fbaf0 0x7f42e40725e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f42e0000c00 tx=0x7f42e0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:10.420 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.416+0000 7f42d2ffd700 1 -- 172.21.15.47:0/88768799 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42e0011690 con 0x7f42e40fbaf0 2024-01-18T06:48:10.420 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.416+0000 7f42d2ffd700 1 -- 172.21.15.47:0/88768799 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f42e000d9f0 con 0x7f42e40fbaf0 2024-01-18T06:48:10.420 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.416+0000 7f42eb175700 1 -- 172.21.15.47:0/88768799 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42e4070c00 con 0x7f42e40fbaf0 2024-01-18T06:48:10.420 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.416+0000 7f42d2ffd700 1 -- 172.21.15.47:0/88768799 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42e001b710 con 0x7f42e40fbaf0 2024-01-18T06:48:10.420 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.416+0000 7f42eb175700 1 -- 172.21.15.47:0/88768799 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42e4071010 con 0x7f42e40fbaf0 2024-01-18T06:48:10.420 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.417+0000 7f42d2ffd700 1 -- 172.21.15.47:0/88768799 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f42e000f070 con 0x7f42e40fbaf0 2024-01-18T06:48:10.420 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.417+0000 7f42eb175700 1 -- 172.21.15.47:0/88768799 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42e4045ad0 con 0x7f42e40fbaf0 2024-01-18T06:48:10.420 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.417+0000 7f42d2ffd700 1 --2- 172.21.15.47:0/88768799 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f42cc02fdb0 0x7f42cc032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:10.420 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.417+0000 7f42d2ffd700 1 -- 172.21.15.47:0/88768799 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f42e0041190 con 0x7f42e40fbaf0 2024-01-18T06:48:10.420 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.417+0000 7f42e9972700 1 --2- 172.21.15.47:0/88768799 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f42cc02fdb0 0x7f42cc032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:10.420 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.418+0000 7f42e9972700 1 --2- 172.21.15.47:0/88768799 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f42cc02fdb0 0x7f42cc032270 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f42d40096a0 tx=0x7f42d4006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:10.422 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.422+0000 7f42d2ffd700 1 -- 172.21.15.47:0/88768799 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f42e0019af0 con 0x7f42e40fbaf0 2024-01-18T06:48:10.609 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.608+0000 7f42eb175700 1 -- 172.21.15.47:0/88768799 --> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f42e4071250 con 0x7f42cc02fdb0 2024-01-18T06:48:10.612 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.612+0000 7f42d2ffd700 1 -- 172.21.15.47:0/88768799 <== mgr.14114 v2:172.21.15.47:6800/3978480908 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f42e4071250 con 0x7f42cc02fdb0 2024-01-18T06:48:10.612 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: Scheduled crash update... 2024-01-18T06:48:10.615 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.614+0000 7f42eb175700 1 -- 172.21.15.47:0/88768799 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f42cc02fdb0 msgr2=0x7f42cc032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:10.615 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.614+0000 7f42eb175700 1 --2- 172.21.15.47:0/88768799 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f42cc02fdb0 0x7f42cc032270 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f42d40096a0 tx=0x7f42d4006b40).stop 2024-01-18T06:48:10.615 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.614+0000 7f42eb175700 1 -- 172.21.15.47:0/88768799 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f42e40fbaf0 msgr2=0x7f42e40725e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:10.615 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.614+0000 7f42eb175700 1 --2- 172.21.15.47:0/88768799 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f42e40fbaf0 0x7f42e40725e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f42e0000c00 tx=0x7f42e0006ee0).stop 2024-01-18T06:48:10.615 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.614+0000 7f42eb175700 1 -- 172.21.15.47:0/88768799 shutdown_connections 2024-01-18T06:48:10.615 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.614+0000 7f42eb175700 1 --2- 172.21.15.47:0/88768799 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f42cc02fdb0 0x7f42cc032270 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:10.615 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.614+0000 7f42eb175700 1 --2- 172.21.15.47:0/88768799 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f42e40fbaf0 0x7f42e40725e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:10.615 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.614+0000 7f42eb175700 1 -- 172.21.15.47:0/88768799 >> 172.21.15.47:0/88768799 conn(0x7f42e40f7350 msgr2=0x7f42e40f8ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:48:10.616 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.614+0000 7f42eb175700 1 -- 172.21.15.47:0/88768799 shutdown_connections 2024-01-18T06:48:10.616 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:10.614+0000 7f42eb175700 1 -- 172.21.15.47:0/88768799 wait complete. 2024-01-18T06:48:10.943 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:10 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:08.780384+0000 mgr.smithi047.fuhzzr (mgr.14114) 10 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-01-18T06:48:10.944 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:10 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cephadm 2024-01-18T06:48:08.781149+0000 mgr.smithi047.fuhzzr (mgr.14114) 11 : cephadm [INF] Saving service mon spec with placement count:5 2024-01-18T06:48:10.944 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:10 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:09.690628+0000 mgr.smithi047.fuhzzr (mgr.14114) 12 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-01-18T06:48:10.944 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:10 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cephadm 2024-01-18T06:48:09.691164+0000 mgr.smithi047.fuhzzr (mgr.14114) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2024-01-18T06:48:10.944 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:10 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:09.693388+0000 mon.smithi047 (mon.0) 57 : audit [INF] from='mgr.14114 172.21.15.47:0/2934400552' entity='mgr.smithi047.fuhzzr' 2024-01-18T06:48:10.944 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:10 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:10.613510+0000 mon.smithi047 (mon.0) 58 : audit [INF] from='mgr.14114 172.21.15.47:0/2934400552' entity='mgr.smithi047.fuhzzr' 2024-01-18T06:48:10.980 INFO:teuthology.orchestra.run.smithi047.stderr:Enabling mgr prometheus module... 2024-01-18T06:48:10.980 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/653e4012-b5cd-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpck5s1jz7:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcg0aieo2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2024-01-18T06:48:11.348 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.343+0000 7f09db59e700 1 Processor -- start 2024-01-18T06:48:11.348 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.344+0000 7f09db59e700 1 -- start start 2024-01-18T06:48:11.348 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.344+0000 7f09db59e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09dc0703f0 0x7f09dc0fe770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:11.348 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.344+0000 7f09db59e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09dc0fec60 con 0x7f09dc0703f0 2024-01-18T06:48:11.348 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.344+0000 7f09da59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09dc0703f0 0x7f09dc0fe770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:11.348 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.344+0000 7f09da59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09dc0703f0 0x7f09dc0fe770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57782/0 (socket says 172.21.15.47:57782) 2024-01-18T06:48:11.348 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.344+0000 7f09da59c700 1 -- 172.21.15.47:0/2659270896 learned_addr learned my addr 172.21.15.47:0/2659270896 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:48:11.349 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.344+0000 7f09da59c700 1 -- 172.21.15.47:0/2659270896 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09dc0feda0 con 0x7f09dc0703f0 2024-01-18T06:48:11.349 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.344+0000 7f09da59c700 1 --2- 172.21.15.47:0/2659270896 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09dc0703f0 0x7f09dc0fe770 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f09c400b5f0 tx=0x7f09c4006ee0).ready entity=mon.0 client_cookie=91c9f4fc7cc06793 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:11.349 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.345+0000 7f09d959a700 1 -- 172.21.15.47:0/2659270896 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09c400d600 con 0x7f09dc0703f0 2024-01-18T06:48:11.349 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.345+0000 7f09d959a700 1 -- 172.21.15.47:0/2659270896 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f09c400dbd0 con 0x7f09dc0703f0 2024-01-18T06:48:11.349 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.345+0000 7f09d959a700 1 -- 172.21.15.47:0/2659270896 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09c4011c00 con 0x7f09dc0703f0 2024-01-18T06:48:11.349 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.345+0000 7f09db59e700 1 -- 172.21.15.47:0/2659270896 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09dc0703f0 msgr2=0x7f09dc0fe770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:11.349 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.345+0000 7f09db59e700 1 --2- 172.21.15.47:0/2659270896 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09dc0703f0 0x7f09dc0fe770 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f09c400b5f0 tx=0x7f09c4006ee0).stop 2024-01-18T06:48:11.349 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.345+0000 7f09db59e700 1 -- 172.21.15.47:0/2659270896 shutdown_connections 2024-01-18T06:48:11.349 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.345+0000 7f09db59e700 1 --2- 172.21.15.47:0/2659270896 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09dc0703f0 0x7f09dc0fe770 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:11.349 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.345+0000 7f09db59e700 1 -- 172.21.15.47:0/2659270896 >> 172.21.15.47:0/2659270896 conn(0x7f09dc0f6210 msgr2=0x7f09dc0f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:48:11.349 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.345+0000 7f09db59e700 1 -- 172.21.15.47:0/2659270896 shutdown_connections 2024-01-18T06:48:11.349 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.345+0000 7f09db59e700 1 -- 172.21.15.47:0/2659270896 wait complete. 2024-01-18T06:48:11.349 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.346+0000 7f09db59e700 1 Processor -- start 2024-01-18T06:48:11.350 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.346+0000 7f09db59e700 1 -- start start 2024-01-18T06:48:11.350 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.346+0000 7f09db59e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09dc0703f0 0x7f09dc0fc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:11.350 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.346+0000 7f09db59e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09dc0fa670 con 0x7f09dc0703f0 2024-01-18T06:48:11.350 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.346+0000 7f09da59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09dc0703f0 0x7f09dc0fc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:11.350 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.346+0000 7f09da59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09dc0703f0 0x7f09dc0fc1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:57796/0 (socket says 172.21.15.47:57796) 2024-01-18T06:48:11.350 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.346+0000 7f09da59c700 1 -- 172.21.15.47:0/1904211207 learned_addr learned my addr 172.21.15.47:0/1904211207 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:48:11.350 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.347+0000 7f09da59c700 1 -- 172.21.15.47:0/1904211207 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c400b200 con 0x7f09dc0703f0 2024-01-18T06:48:11.350 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.347+0000 7f09da59c700 1 --2- 172.21.15.47:0/1904211207 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09dc0703f0 0x7f09dc0fc1f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f09c400e040 tx=0x7f09c4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:11.350 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.347+0000 7f09d2ffd700 1 -- 172.21.15.47:0/1904211207 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09c400d9f0 con 0x7f09dc0703f0 2024-01-18T06:48:11.350 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.347+0000 7f09d2ffd700 1 -- 172.21.15.47:0/1904211207 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f09c401a9f0 con 0x7f09dc0703f0 2024-01-18T06:48:11.350 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.347+0000 7f09db59e700 1 -- 172.21.15.47:0/1904211207 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09dc0fa810 con 0x7f09dc0703f0 2024-01-18T06:48:11.350 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.347+0000 7f09d2ffd700 1 -- 172.21.15.47:0/1904211207 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09c401b8c0 con 0x7f09dc0703f0 2024-01-18T06:48:11.350 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.347+0000 7f09db59e700 1 -- 172.21.15.47:0/1904211207 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09dc0fac20 con 0x7f09dc0703f0 2024-01-18T06:48:11.350 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.348+0000 7f09d2ffd700 1 -- 172.21.15.47:0/1904211207 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f09c400f070 con 0x7f09dc0703f0 2024-01-18T06:48:11.351 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.348+0000 7f09d2ffd700 1 --2- 172.21.15.47:0/1904211207 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f09bc02fdb0 0x7f09bc032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:11.351 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.348+0000 7f09d2ffd700 1 -- 172.21.15.47:0/1904211207 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f09c40412d0 con 0x7f09dc0703f0 2024-01-18T06:48:11.351 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.348+0000 7f09db59e700 1 -- 172.21.15.47:0/1904211207 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09dc045ad0 con 0x7f09dc0703f0 2024-01-18T06:48:11.352 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.348+0000 7f09d9d9b700 1 --2- 172.21.15.47:0/1904211207 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f09bc02fdb0 0x7f09bc032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:11.352 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.349+0000 7f09d9d9b700 1 --2- 172.21.15.47:0/1904211207 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f09bc02fdb0 0x7f09bc032270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f09cc0096a0 tx=0x7f09cc006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:11.353 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.352+0000 7f09d2ffd700 1 -- 172.21.15.47:0/1904211207 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f09c4016070 con 0x7f09dc0703f0 2024-01-18T06:48:11.548 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.547+0000 7f09db59e700 1 -- 172.21.15.47:0/1904211207 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7f09dc0ffa20 con 0x7f09dc0703f0 2024-01-18T06:48:11.693 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.692+0000 7f09d2ffd700 1 -- 172.21.15.47:0/1904211207 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f09c40257e0 con 0x7f09dc0703f0 2024-01-18T06:48:11.694 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.693+0000 7f09d2ffd700 1 -- 172.21.15.47:0/1904211207 <== mon.0 v2:172.21.15.47:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v9) v1 ==== 89+0+0 (secure 0 0 0) 0x7f09c4025460 con 0x7f09dc0703f0 2024-01-18T06:48:11.697 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.695+0000 7f09db59e700 1 -- 172.21.15.47:0/1904211207 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f09bc02fdb0 msgr2=0x7f09bc032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:11.697 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.695+0000 7f09db59e700 1 --2- 172.21.15.47:0/1904211207 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f09bc02fdb0 0x7f09bc032270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f09cc0096a0 tx=0x7f09cc006b40).stop 2024-01-18T06:48:11.697 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.695+0000 7f09db59e700 1 -- 172.21.15.47:0/1904211207 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09dc0703f0 msgr2=0x7f09dc0fc1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:11.697 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.695+0000 7f09db59e700 1 --2- 172.21.15.47:0/1904211207 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09dc0703f0 0x7f09dc0fc1f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f09c400e040 tx=0x7f09c4006ee0).stop 2024-01-18T06:48:11.697 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.696+0000 7f09db59e700 1 -- 172.21.15.47:0/1904211207 shutdown_connections 2024-01-18T06:48:11.697 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.696+0000 7f09db59e700 1 --2- 172.21.15.47:0/1904211207 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f09bc02fdb0 0x7f09bc032270 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:11.697 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.696+0000 7f09db59e700 1 --2- 172.21.15.47:0/1904211207 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f09dc0703f0 0x7f09dc0fc1f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:11.697 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.696+0000 7f09db59e700 1 -- 172.21.15.47:0/1904211207 >> 172.21.15.47:0/1904211207 conn(0x7f09dc0f6210 msgr2=0x7f09dc0f7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:48:11.697 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.696+0000 7f09db59e700 1 -- 172.21.15.47:0/1904211207 shutdown_connections 2024-01-18T06:48:11.697 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:11.696+0000 7f09db59e700 1 -- 172.21.15.47:0/1904211207 wait complete. 2024-01-18T06:48:11.963 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:11 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:10.610641+0000 mgr.smithi047.fuhzzr (mgr.14114) 14 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-01-18T06:48:11.963 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:11 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cephadm 2024-01-18T06:48:10.611310+0000 mgr.smithi047.fuhzzr (mgr.14114) 15 : cephadm [INF] Saving service crash spec with placement * 2024-01-18T06:48:11.964 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:11 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:11.549275+0000 mon.smithi047 (mon.0) 59 : audit [INF] from='client.? 172.21.15.47:0/1904211207' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-01-18T06:48:12.172 INFO:teuthology.orchestra.run.smithi047.stderr:Deploying prometheus service with default placement... 2024-01-18T06:48:12.172 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/653e4012-b5cd-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpck5s1jz7:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcg0aieo2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2024-01-18T06:48:12.549 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:12.547+0000 7f927a427700 1 Processor -- start 2024-01-18T06:48:12.549 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:12.547+0000 7f927a427700 1 -- start start 2024-01-18T06:48:12.549 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:12.548+0000 7f927a427700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9274070430 0x7f92740fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:12.549 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:12.548+0000 7f927a427700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92740fedd0 con 0x7f9274070430 2024-01-18T06:48:12.549 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:12.548+0000 7f9273fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9274070430 0x7f92740fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:12.549 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:12.548+0000 7f9273fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9274070430 0x7f92740fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60616/0 (socket says 172.21.15.47:60616) 2024-01-18T06:48:12.550 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:12.548+0000 7f9273fff700 1 -- 172.21.15.47:0/3055258250 learned_addr learned my addr 172.21.15.47:0/3055258250 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:48:12.550 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:12.548+0000 7f9273fff700 1 -- 172.21.15.47:0/3055258250 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92740fef10 con 0x7f9274070430 2024-01-18T06:48:12.550 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:12.548+0000 7f9273fff700 1 --2- 172.21.15.47:0/3055258250 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9274070430 0x7f92740fe8e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f926400b5f0 tx=0x7f9264006ee0).ready entity=mon.0 client_cookie=91dd825203ba02de server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:12.550 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:12.548+0000 7f9272ffd700 1 -- 172.21.15.47:0/3055258250 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f926400d600 con 0x7f9274070430 2024-01-18T06:48:12.550 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:12.548+0000 7f9272ffd700 1 -- 172.21.15.47:0/3055258250 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f926400dbd0 con 0x7f9274070430 2024-01-18T06:48:12.551 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:12.548+0000 7f9272ffd700 1 -- 172.21.15.47:0/3055258250 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9264011c00 con 0x7f9274070430 2024-01-18T06:48:12.551 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:12.549+0000 7f927a427700 1 -- 172.21.15.47:0/3055258250 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9274070430 msgr2=0x7f92740fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:12.551 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:12.549+0000 7f927a427700 1 --2- 172.21.15.47:0/3055258250 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9274070430 0x7f92740fe8e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f926400b5f0 tx=0x7f9264006ee0).stop 2024-01-18T06:48:12.552 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:12.549+0000 7f927a427700 1 -- 172.21.15.47:0/3055258250 shutdown_connections 2024-01-18T06:48:12.552 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:12.549+0000 7f927a427700 1 --2- 172.21.15.47:0/3055258250 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9274070430 0x7f92740fe8e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:12.552 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:12.549+0000 7f927a427700 1 -- 172.21.15.47:0/3055258250 >> 172.21.15.47:0/3055258250 conn(0x7f92740f63c0 msgr2=0x7f92740f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:48:12.552 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:12.549+0000 7f927a427700 1 -- 172.21.15.47:0/3055258250 shutdown_connections 2024-01-18T06:48:12.552 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:12.549+0000 7f927a427700 1 -- 172.21.15.47:0/3055258250 wait complete. 2024-01-18T06:48:12.552 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:12.550+0000 7f927a427700 1 Processor -- start 2024-01-18T06:48:12.552 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:12.550+0000 7f927a427700 1 -- start start 2024-01-18T06:48:12.552 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:12.550+0000 7f927a427700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9274070430 0x7f927410aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:12.552 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:12.550+0000 7f927a427700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f927410aef0 con 0x7f9274070430 2024-01-18T06:48:12.552 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:12.550+0000 7f9273fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9274070430 0x7f927410aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:12.552 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:12.550+0000 7f9273fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9274070430 0x7f927410aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60622/0 (socket says 172.21.15.47:60622) 2024-01-18T06:48:12.552 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:12.550+0000 7f9273fff700 1 -- 172.21.15.47:0/2867837293 learned_addr learned my addr 172.21.15.47:0/2867837293 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:48:12.553 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:12.551+0000 7f9273fff700 1 -- 172.21.15.47:0/2867837293 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f926400b200 con 0x7f9274070430 2024-01-18T06:48:12.553 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:12.551+0000 7f9273fff700 1 --2- 172.21.15.47:0/2867837293 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9274070430 0x7f927410aa00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f926400b190 tx=0x7f9264006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:12.554 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:12.551+0000 7f9270ff9700 1 -- 172.21.15.47:0/2867837293 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f926401b410 con 0x7f9274070430 2024-01-18T06:48:12.554 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:12.551+0000 7f9270ff9700 1 -- 172.21.15.47:0/2867837293 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f926401b9e0 con 0x7f9274070430 2024-01-18T06:48:12.554 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:12.551+0000 7f927a427700 1 -- 172.21.15.47:0/2867837293 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f927410b0f0 con 0x7f9274070430 2024-01-18T06:48:12.554 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:12.551+0000 7f9270ff9700 1 -- 172.21.15.47:0/2867837293 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f926401a880 con 0x7f9274070430 2024-01-18T06:48:12.554 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:12.551+0000 7f927a427700 1 -- 172.21.15.47:0/2867837293 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f927410b500 con 0x7f9274070430 2024-01-18T06:48:12.554 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:12.552+0000 7f9270ff9700 1 -- 172.21.15.47:0/2867837293 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f926400f070 con 0x7f9274070430 2024-01-18T06:48:12.554 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:12.552+0000 7f9270ff9700 1 --2- 172.21.15.47:0/2867837293 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f926002fe00 0x7f92600322c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:12.554 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:12.552+0000 7f9270ff9700 1 -- 172.21.15.47:0/2867837293 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f92640413c0 con 0x7f9274070430 2024-01-18T06:48:12.554 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:12.552+0000 7f92737fe700 1 -- 172.21.15.47:0/2867837293 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f926002fe00 msgr2=0x7f92600322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/3978480908 2024-01-18T06:48:12.554 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:12.552+0000 7f92737fe700 1 --2- 172.21.15.47:0/2867837293 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f926002fe00 0x7f92600322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-18T06:48:12.554 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:12.553+0000 7f927a427700 1 -- 172.21.15.47:0/2867837293 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9274059100 con 0x7f9274070430 2024-01-18T06:48:12.557 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:12.557+0000 7f9270ff9700 1 -- 172.21.15.47:0/2867837293 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9264016070 con 0x7f9274070430 2024-01-18T06:48:12.745 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:12.745+0000 7f927a427700 1 -- 172.21.15.47:0/2867837293 --> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f9274104860 con 0x7f926002fe00 2024-01-18T06:48:12.753 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:12.753+0000 7f92737fe700 1 -- 172.21.15.47:0/2867837293 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f926002fe00 msgr2=0x7f92600322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/3978480908 2024-01-18T06:48:12.753 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:12.753+0000 7f92737fe700 1 --2- 172.21.15.47:0/2867837293 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f926002fe00 0x7f92600322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-18T06:48:12.963 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:12 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:11.695319+0000 mon.smithi047 (mon.0) 60 : audit [INF] from='client.? 172.21.15.47:0/1904211207' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-01-18T06:48:12.963 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:12 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:48:11.695383+0000 mon.smithi047 (mon.0) 61 : cluster [DBG] mgrmap e9: smithi047.fuhzzr(active, since 18s) 2024-01-18T06:48:13.154 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:13.154+0000 7f92737fe700 1 -- 172.21.15.47:0/2867837293 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f926002fe00 msgr2=0x7f92600322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/3978480908 2024-01-18T06:48:13.154 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:13.154+0000 7f92737fe700 1 --2- 172.21.15.47:0/2867837293 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f926002fe00 0x7f92600322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-01-18T06:48:13.955 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:13.955+0000 7f92737fe700 1 -- 172.21.15.47:0/2867837293 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f926002fe00 msgr2=0x7f92600322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/3978480908 2024-01-18T06:48:13.956 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:13.955+0000 7f92737fe700 1 --2- 172.21.15.47:0/2867837293 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f926002fe00 0x7f92600322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-01-18T06:48:15.557 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:15.556+0000 7f92737fe700 1 -- 172.21.15.47:0/2867837293 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f926002fe00 msgr2=0x7f92600322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/3978480908 2024-01-18T06:48:15.557 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:15.556+0000 7f92737fe700 1 --2- 172.21.15.47:0/2867837293 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f926002fe00 0x7f92600322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-01-18T06:48:16.918 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:16.917+0000 7f9270ff9700 1 -- 172.21.15.47:0/2867837293 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mgrmap(e 10) v1 ==== 35984+0+0 (secure 0 0 0) 0x7f92640257e0 con 0x7f9274070430 2024-01-18T06:48:16.919 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:16.917+0000 7f9270ff9700 1 -- 172.21.15.47:0/2867837293 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f926002fe00 msgr2=0x7f92600322c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-18T06:48:16.919 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:16.917+0000 7f9270ff9700 1 --2- 172.21.15.47:0/2867837293 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f926002fe00 0x7f92600322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-18T06:48:17.920 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:17.919+0000 7f9270ff9700 1 -- 172.21.15.47:0/2867837293 <== mon.0 v2:172.21.15.47:3300/0 8 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f9264043150 con 0x7f9274070430 2024-01-18T06:48:17.920 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:17.919+0000 7f9270ff9700 1 --2- 172.21.15.47:0/2867837293 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f9260033c20 0x7f9260036010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:17.921 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:17.919+0000 7f9270ff9700 1 -- 172.21.15.47:0/2867837293 --> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f9274104860 con 0x7f9260033c20 2024-01-18T06:48:17.921 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:17.919+0000 7f92737fe700 1 --2- 172.21.15.47:0/2867837293 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f9260033c20 0x7f9260036010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:17.921 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:17.920+0000 7f92737fe700 1 --2- 172.21.15.47:0/2867837293 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f9260033c20 0x7f9260036010 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f92680036d0 tx=0x7f9268006ce0).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:17.925 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:17.925+0000 7f9270ff9700 1 -- 172.21.15.47:0/2867837293 <== mgr.14142 v2:172.21.15.47:6800/1162118152 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f9274104860 con 0x7f9260033c20 2024-01-18T06:48:17.925 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-01-18T06:48:17.928 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:17.927+0000 7f927a427700 1 -- 172.21.15.47:0/2867837293 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f9260033c20 msgr2=0x7f9260036010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:17.929 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:17.927+0000 7f927a427700 1 --2- 172.21.15.47:0/2867837293 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f9260033c20 0x7f9260036010 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f92680036d0 tx=0x7f9268006ce0).stop 2024-01-18T06:48:17.929 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:17.927+0000 7f927a427700 1 -- 172.21.15.47:0/2867837293 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9274070430 msgr2=0x7f927410aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:17.929 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:17.927+0000 7f927a427700 1 --2- 172.21.15.47:0/2867837293 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9274070430 0x7f927410aa00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f926400b190 tx=0x7f9264006ee0).stop 2024-01-18T06:48:17.929 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:17.927+0000 7f927a427700 1 -- 172.21.15.47:0/2867837293 shutdown_connections 2024-01-18T06:48:17.929 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:17.927+0000 7f927a427700 1 --2- 172.21.15.47:0/2867837293 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9274070430 0x7f927410aa00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:17.929 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:17.927+0000 7f927a427700 1 --2- 172.21.15.47:0/2867837293 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f9260033c20 0x7f9260036010 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:17.929 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:17.927+0000 7f927a427700 1 --2- 172.21.15.47:0/2867837293 >> [v2:172.21.15.47:6800/3978480908,v1:172.21.15.47:6801/3978480908] conn(0x7f926002fe00 0x7f92600322c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-18T06:48:17.929 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:17.927+0000 7f927a427700 1 -- 172.21.15.47:0/2867837293 >> 172.21.15.47:0/2867837293 conn(0x7f92740f63c0 msgr2=0x7f92740f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:48:17.929 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:17.927+0000 7f927a427700 1 -- 172.21.15.47:0/2867837293 shutdown_connections 2024-01-18T06:48:17.929 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:17.927+0000 7f927a427700 1 -- 172.21.15.47:0/2867837293 wait complete. 2024-01-18T06:48:18.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:17 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:48:16.867492+0000 mon.smithi047 (mon.0) 62 : cluster [INF] Active manager daemon smithi047.fuhzzr restarted 2024-01-18T06:48:18.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:17 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:48:16.867927+0000 mon.smithi047 (mon.0) 63 : cluster [INF] Activating manager daemon smithi047.fuhzzr 2024-01-18T06:48:18.214 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:17 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:48:16.869592+0000 mon.smithi047 (mon.0) 64 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-01-18T06:48:18.214 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:17 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:48:16.920577+0000 mon.smithi047 (mon.0) 65 : cluster [DBG] mgrmap e10: smithi047.fuhzzr(active, starting, since 0.0527676s) 2024-01-18T06:48:18.214 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:17 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:16.921226+0000 mon.smithi047 (mon.0) 66 : audit [DBG] from='mgr.14142 172.21.15.47:0/122516226' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix": "mon metadata", "id": "smithi047"}]: dispatch 2024-01-18T06:48:18.214 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:17 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:16.921876+0000 mon.smithi047 (mon.0) 67 : audit [DBG] from='mgr.14142 172.21.15.47:0/122516226' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix": "mgr metadata", "who": "smithi047.fuhzzr", "id": "smithi047.fuhzzr"}]: dispatch 2024-01-18T06:48:18.214 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:17 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:16.922727+0000 mon.smithi047 (mon.0) 68 : audit [DBG] from='mgr.14142 172.21.15.47:0/122516226' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-18T06:48:18.214 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:17 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:16.922881+0000 mon.smithi047 (mon.0) 69 : audit [DBG] from='mgr.14142 172.21.15.47:0/122516226' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-18T06:48:18.214 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:17 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:16.923003+0000 mon.smithi047 (mon.0) 70 : audit [DBG] from='mgr.14142 172.21.15.47:0/122516226' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-18T06:48:18.214 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:17 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:48:16.937179+0000 mon.smithi047 (mon.0) 71 : cluster [INF] Manager daemon smithi047.fuhzzr is now available 2024-01-18T06:48:18.214 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:17 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:17.039749+0000 mon.smithi047 (mon.0) 72 : audit [DBG] from='mgr.14142 172.21.15.47:0/122516226' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-18T06:48:18.214 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:17 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:17.041361+0000 mon.smithi047 (mon.0) 73 : audit [DBG] from='mgr.14142 172.21.15.47:0/122516226' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-18T06:48:18.214 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:17 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:17.057090+0000 mon.smithi047 (mon.0) 74 : audit [INF] from='mgr.14142 172.21.15.47:0/122516226' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.fuhzzr/mirror_snapshot_schedule"}]: dispatch 2024-01-18T06:48:18.214 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:17 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:17.059579+0000 mon.smithi047 (mon.0) 75 : audit [INF] from='mgr.14142 172.21.15.47:0/122516226' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.fuhzzr/trash_purge_schedule"}]: dispatch 2024-01-18T06:48:18.248 INFO:teuthology.orchestra.run.smithi047.stderr:Deploying grafana service with default placement... 2024-01-18T06:48:18.248 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/653e4012-b5cd-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpck5s1jz7:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcg0aieo2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2024-01-18T06:48:18.605 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.603+0000 7f1ee756c700 1 Processor -- start 2024-01-18T06:48:18.605 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.603+0000 7f1ee756c700 1 -- start start 2024-01-18T06:48:18.605 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.604+0000 7f1ee756c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ee00703f0 0x7f1ee00fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:18.605 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.604+0000 7f1ee756c700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee00fee10 con 0x7f1ee00703f0 2024-01-18T06:48:18.606 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.604+0000 7f1ee5308700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ee00703f0 0x7f1ee00fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:18.606 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.604+0000 7f1ee5308700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ee00703f0 0x7f1ee00fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60688/0 (socket says 172.21.15.47:60688) 2024-01-18T06:48:18.606 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.604+0000 7f1ee5308700 1 -- 172.21.15.47:0/1140415273 learned_addr learned my addr 172.21.15.47:0/1140415273 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:48:18.606 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.604+0000 7f1ee5308700 1 -- 172.21.15.47:0/1140415273 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee00fef50 con 0x7f1ee00703f0 2024-01-18T06:48:18.606 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.605+0000 7f1ee5308700 1 --2- 172.21.15.47:0/1140415273 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ee00703f0 0x7f1ee00fe920 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f1edc00b5f0 tx=0x7f1edc006ee0).ready entity=mon.0 client_cookie=df6397068915b2a8 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:18.606 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.605+0000 7f1ecffff700 1 -- 172.21.15.47:0/1140415273 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1edc00d600 con 0x7f1ee00703f0 2024-01-18T06:48:18.606 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.605+0000 7f1ecffff700 1 -- 172.21.15.47:0/1140415273 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1edc00dbd0 con 0x7f1ee00703f0 2024-01-18T06:48:18.606 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.605+0000 7f1ecffff700 1 -- 172.21.15.47:0/1140415273 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1edc011c00 con 0x7f1ee00703f0 2024-01-18T06:48:18.608 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.605+0000 7f1ee756c700 1 -- 172.21.15.47:0/1140415273 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ee00703f0 msgr2=0x7f1ee00fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:18.608 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.605+0000 7f1ee756c700 1 --2- 172.21.15.47:0/1140415273 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ee00703f0 0x7f1ee00fe920 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f1edc00b5f0 tx=0x7f1edc006ee0).stop 2024-01-18T06:48:18.608 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.605+0000 7f1ee756c700 1 -- 172.21.15.47:0/1140415273 shutdown_connections 2024-01-18T06:48:18.608 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.605+0000 7f1ee756c700 1 --2- 172.21.15.47:0/1140415273 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ee00703f0 0x7f1ee00fe920 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:18.609 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.605+0000 7f1ee756c700 1 -- 172.21.15.47:0/1140415273 >> 172.21.15.47:0/1140415273 conn(0x7f1ee00f63c0 msgr2=0x7f1ee00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:48:18.609 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.606+0000 7f1ee756c700 1 -- 172.21.15.47:0/1140415273 shutdown_connections 2024-01-18T06:48:18.609 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.606+0000 7f1ee756c700 1 -- 172.21.15.47:0/1140415273 wait complete. 2024-01-18T06:48:18.609 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.606+0000 7f1ee756c700 1 Processor -- start 2024-01-18T06:48:18.609 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.606+0000 7f1ee756c700 1 -- start start 2024-01-18T06:48:18.609 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.607+0000 7f1ee756c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ee00703f0 0x7f1ee00fc330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:18.609 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.607+0000 7f1ee756c700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee00fa820 con 0x7f1ee00703f0 2024-01-18T06:48:18.609 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.607+0000 7f1ee5308700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ee00703f0 0x7f1ee00fc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:18.609 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.607+0000 7f1ee5308700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ee00703f0 0x7f1ee00fc330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60696/0 (socket says 172.21.15.47:60696) 2024-01-18T06:48:18.609 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.607+0000 7f1ee5308700 1 -- 172.21.15.47:0/28508212 learned_addr learned my addr 172.21.15.47:0/28508212 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:48:18.609 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.607+0000 7f1ee5308700 1 -- 172.21.15.47:0/28508212 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1edc00b200 con 0x7f1ee00703f0 2024-01-18T06:48:18.609 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.607+0000 7f1ee5308700 1 --2- 172.21.15.47:0/28508212 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ee00703f0 0x7f1ee00fc330 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f1edc00e040 tx=0x7f1edc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:18.609 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.607+0000 7f1ecdffb700 1 -- 172.21.15.47:0/28508212 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1edc00d9f0 con 0x7f1ee00703f0 2024-01-18T06:48:18.610 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.607+0000 7f1ecdffb700 1 -- 172.21.15.47:0/28508212 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1edc01a9f0 con 0x7f1ee00703f0 2024-01-18T06:48:18.610 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.607+0000 7f1ee756c700 1 -- 172.21.15.47:0/28508212 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ee00faa20 con 0x7f1ee00703f0 2024-01-18T06:48:18.610 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.607+0000 7f1ecdffb700 1 -- 172.21.15.47:0/28508212 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1edc01b8c0 con 0x7f1ee00703f0 2024-01-18T06:48:18.610 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.608+0000 7f1ee756c700 1 -- 172.21.15.47:0/28508212 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ee00fae30 con 0x7f1ee00703f0 2024-01-18T06:48:18.611 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.609+0000 7f1ecdffb700 1 -- 172.21.15.47:0/28508212 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f1edc00f070 con 0x7f1ee00703f0 2024-01-18T06:48:18.611 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.609+0000 7f1ee756c700 1 -- 172.21.15.47:0/28508212 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ee0045ad0 con 0x7f1ee00703f0 2024-01-18T06:48:18.611 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.609+0000 7f1ecdffb700 1 --2- 172.21.15.47:0/28508212 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f1ec802fd70 0x7f1ec8032230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:18.611 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.609+0000 7f1ecdffb700 1 -- 172.21.15.47:0/28508212 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f1edc0412e0 con 0x7f1ee00703f0 2024-01-18T06:48:18.611 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.609+0000 7f1ee4b07700 1 --2- 172.21.15.47:0/28508212 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f1ec802fd70 0x7f1ec8032230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:18.611 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.609+0000 7f1ee4b07700 1 --2- 172.21.15.47:0/28508212 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f1ec802fd70 0x7f1ec8032230 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f1ed00096a0 tx=0x7f1ed0006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:18.613 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.613+0000 7f1ecdffb700 1 -- 172.21.15.47:0/28508212 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1edc01a760 con 0x7f1ee00703f0 2024-01-18T06:48:18.802 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.802+0000 7f1ee756c700 1 -- 172.21.15.47:0/28508212 --> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ee00fb130 con 0x7f1ec802fd70 2024-01-18T06:48:18.806 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.806+0000 7f1ecdffb700 1 -- 172.21.15.47:0/28508212 <== mgr.14142 v2:172.21.15.47:6800/1162118152 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f1ee00fb130 con 0x7f1ec802fd70 2024-01-18T06:48:18.806 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-01-18T06:48:18.809 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.807+0000 7f1ee756c700 1 -- 172.21.15.47:0/28508212 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f1ec802fd70 msgr2=0x7f1ec8032230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:18.809 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.807+0000 7f1ee756c700 1 --2- 172.21.15.47:0/28508212 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f1ec802fd70 0x7f1ec8032230 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f1ed00096a0 tx=0x7f1ed0006b40).stop 2024-01-18T06:48:18.809 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.807+0000 7f1ee756c700 1 -- 172.21.15.47:0/28508212 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ee00703f0 msgr2=0x7f1ee00fc330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:18.809 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.807+0000 7f1ee756c700 1 --2- 172.21.15.47:0/28508212 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ee00703f0 0x7f1ee00fc330 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f1edc00e040 tx=0x7f1edc006ee0).stop 2024-01-18T06:48:18.809 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.807+0000 7f1ee756c700 1 -- 172.21.15.47:0/28508212 shutdown_connections 2024-01-18T06:48:18.810 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.807+0000 7f1ee756c700 1 --2- 172.21.15.47:0/28508212 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f1ec802fd70 0x7f1ec8032230 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:18.810 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.807+0000 7f1ee756c700 1 --2- 172.21.15.47:0/28508212 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ee00703f0 0x7f1ee00fc330 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:18.810 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.807+0000 7f1ee756c700 1 -- 172.21.15.47:0/28508212 >> 172.21.15.47:0/28508212 conn(0x7f1ee00f63c0 msgr2=0x7f1ee00f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:48:18.810 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.807+0000 7f1ee756c700 1 -- 172.21.15.47:0/28508212 shutdown_connections 2024-01-18T06:48:18.810 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:18.807+0000 7f1ee756c700 1 -- 172.21.15.47:0/28508212 wait complete. 2024-01-18T06:48:19.107 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:18 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:48:17.922568+0000 mon.smithi047 (mon.0) 76 : cluster [DBG] mgrmap e11: smithi047.fuhzzr(active, since 1.05476s) 2024-01-18T06:48:19.107 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:18 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cephadm 2024-01-18T06:48:17.923979+0000 mgr.smithi047.fuhzzr (mgr.14142) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2024-01-18T06:48:19.107 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:18 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:17.926192+0000 mon.smithi047 (mon.0) 77 : audit [INF] from='mgr.14142 172.21.15.47:0/122516226' entity='mgr.smithi047.fuhzzr' 2024-01-18T06:48:19.107 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:18 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:18.807222+0000 mon.smithi047 (mon.0) 78 : audit [INF] from='mgr.14142 172.21.15.47:0/122516226' entity='mgr.smithi047.fuhzzr' 2024-01-18T06:48:19.163 INFO:teuthology.orchestra.run.smithi047.stderr:Deploying node-exporter service with default placement... 2024-01-18T06:48:19.163 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/653e4012-b5cd-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpck5s1jz7:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcg0aieo2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2024-01-18T06:48:19.529 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.525+0000 7f7779810700 1 Processor -- start 2024-01-18T06:48:19.530 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.525+0000 7f7779810700 1 -- start start 2024-01-18T06:48:19.530 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.525+0000 7f7779810700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7774070390 0x7f7774100d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:19.530 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.525+0000 7f7779810700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77740fc770 con 0x7f7774070390 2024-01-18T06:48:19.530 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.525+0000 7f7772ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7774070390 0x7f7774100d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:19.530 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.525+0000 7f7772ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7774070390 0x7f7774100d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60708/0 (socket says 172.21.15.47:60708) 2024-01-18T06:48:19.530 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.525+0000 7f7772ffd700 1 -- 172.21.15.47:0/3196607927 learned_addr learned my addr 172.21.15.47:0/3196607927 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:48:19.530 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.526+0000 7f7772ffd700 1 -- 172.21.15.47:0/3196607927 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7774101270 con 0x7f7774070390 2024-01-18T06:48:19.530 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.526+0000 7f7772ffd700 1 --2- 172.21.15.47:0/3196607927 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7774070390 0x7f7774100d80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f775c00b5f0 tx=0x7f775c006ee0).ready entity=mon.0 client_cookie=fbc729c6b1f6bcec server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:19.530 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.526+0000 7f7771ffb700 1 -- 172.21.15.47:0/3196607927 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f775c00d600 con 0x7f7774070390 2024-01-18T06:48:19.530 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.526+0000 7f7771ffb700 1 -- 172.21.15.47:0/3196607927 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f775c00dbd0 con 0x7f7774070390 2024-01-18T06:48:19.530 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.526+0000 7f7771ffb700 1 -- 172.21.15.47:0/3196607927 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f775c011c00 con 0x7f7774070390 2024-01-18T06:48:19.531 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.527+0000 7f7779810700 1 -- 172.21.15.47:0/3196607927 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7774070390 msgr2=0x7f7774100d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:19.531 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.527+0000 7f7779810700 1 --2- 172.21.15.47:0/3196607927 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7774070390 0x7f7774100d80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f775c00b5f0 tx=0x7f775c006ee0).stop 2024-01-18T06:48:19.531 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.527+0000 7f7779810700 1 -- 172.21.15.47:0/3196607927 shutdown_connections 2024-01-18T06:48:19.531 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.527+0000 7f7779810700 1 --2- 172.21.15.47:0/3196607927 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7774070390 0x7f7774100d80 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:19.531 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.527+0000 7f7779810700 1 -- 172.21.15.47:0/3196607927 >> 172.21.15.47:0/3196607927 conn(0x7f77740f63c0 msgr2=0x7f77740f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:48:19.531 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.527+0000 7f7779810700 1 -- 172.21.15.47:0/3196607927 shutdown_connections 2024-01-18T06:48:19.531 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.527+0000 7f7779810700 1 -- 172.21.15.47:0/3196607927 wait complete. 2024-01-18T06:48:19.531 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.528+0000 7f7779810700 1 Processor -- start 2024-01-18T06:48:19.531 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.528+0000 7f7779810700 1 -- start start 2024-01-18T06:48:19.531 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.528+0000 7f7779810700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7774070390 0x7f777410ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:19.531 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.528+0000 7f7779810700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f777410d1b0 con 0x7f7774070390 2024-01-18T06:48:19.531 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.528+0000 7f7772ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7774070390 0x7f777410ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:19.531 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.528+0000 7f7772ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7774070390 0x7f777410ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60722/0 (socket says 172.21.15.47:60722) 2024-01-18T06:48:19.531 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.528+0000 7f7772ffd700 1 -- 172.21.15.47:0/693196991 learned_addr learned my addr 172.21.15.47:0/693196991 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:48:19.532 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.528+0000 7f7772ffd700 1 -- 172.21.15.47:0/693196991 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f775c00b200 con 0x7f7774070390 2024-01-18T06:48:19.532 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.528+0000 7f7772ffd700 1 --2- 172.21.15.47:0/693196991 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7774070390 0x7f777410ccc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f775c00b190 tx=0x7f775c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:19.532 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.529+0000 7f776b7fe700 1 -- 172.21.15.47:0/693196991 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f775c01b410 con 0x7f7774070390 2024-01-18T06:48:19.532 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.529+0000 7f776b7fe700 1 -- 172.21.15.47:0/693196991 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f775c01b9e0 con 0x7f7774070390 2024-01-18T06:48:19.532 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.529+0000 7f776b7fe700 1 -- 172.21.15.47:0/693196991 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f775c01a880 con 0x7f7774070390 2024-01-18T06:48:19.532 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.529+0000 7f7779810700 1 -- 172.21.15.47:0/693196991 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f777410d3b0 con 0x7f7774070390 2024-01-18T06:48:19.532 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.529+0000 7f7779810700 1 -- 172.21.15.47:0/693196991 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f777410d7c0 con 0x7f7774070390 2024-01-18T06:48:19.533 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.529+0000 7f7779810700 1 -- 172.21.15.47:0/693196991 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7774045ad0 con 0x7f7774070390 2024-01-18T06:48:19.533 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.530+0000 7f776b7fe700 1 -- 172.21.15.47:0/693196991 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f775c00f070 con 0x7f7774070390 2024-01-18T06:48:19.533 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.530+0000 7f776b7fe700 1 --2- 172.21.15.47:0/693196991 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f775402fcf0 0x7f77540321b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:19.533 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.530+0000 7f776b7fe700 1 -- 172.21.15.47:0/693196991 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f775c041300 con 0x7f7774070390 2024-01-18T06:48:19.533 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.530+0000 7f77727fc700 1 --2- 172.21.15.47:0/693196991 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f775402fcf0 0x7f77540321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:19.533 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.530+0000 7f77727fc700 1 --2- 172.21.15.47:0/693196991 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f775402fcf0 0x7f77540321b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f77640096a0 tx=0x7f7764006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:19.534 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.534+0000 7f776b7fe700 1 -- 172.21.15.47:0/693196991 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f775c0224f0 con 0x7f7774070390 2024-01-18T06:48:19.719 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.719+0000 7f7779810700 1 -- 172.21.15.47:0/693196991 --> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f777410def0 con 0x7f775402fcf0 2024-01-18T06:48:19.723 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.723+0000 7f776b7fe700 1 -- 172.21.15.47:0/693196991 <== mgr.14142 v2:172.21.15.47:6800/1162118152 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f777410def0 con 0x7f775402fcf0 2024-01-18T06:48:19.723 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-01-18T06:48:19.726 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.724+0000 7f7779810700 1 -- 172.21.15.47:0/693196991 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f775402fcf0 msgr2=0x7f77540321b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:19.726 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.724+0000 7f7779810700 1 --2- 172.21.15.47:0/693196991 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f775402fcf0 0x7f77540321b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f77640096a0 tx=0x7f7764006b40).stop 2024-01-18T06:48:19.727 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.725+0000 7f7779810700 1 -- 172.21.15.47:0/693196991 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7774070390 msgr2=0x7f777410ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:19.727 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.725+0000 7f7779810700 1 --2- 172.21.15.47:0/693196991 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7774070390 0x7f777410ccc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f775c00b190 tx=0x7f775c006ee0).stop 2024-01-18T06:48:19.727 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.725+0000 7f7779810700 1 -- 172.21.15.47:0/693196991 shutdown_connections 2024-01-18T06:48:19.727 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.725+0000 7f7779810700 1 --2- 172.21.15.47:0/693196991 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f775402fcf0 0x7f77540321b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:19.727 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.725+0000 7f7779810700 1 --2- 172.21.15.47:0/693196991 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7774070390 0x7f777410ccc0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:19.727 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.725+0000 7f7779810700 1 -- 172.21.15.47:0/693196991 >> 172.21.15.47:0/693196991 conn(0x7f77740f63c0 msgr2=0x7f77740f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:48:19.727 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.725+0000 7f7779810700 1 -- 172.21.15.47:0/693196991 shutdown_connections 2024-01-18T06:48:19.727 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:19.725+0000 7f7779810700 1 -- 172.21.15.47:0/693196991 wait complete. 2024-01-18T06:48:20.087 INFO:teuthology.orchestra.run.smithi047.stderr:Deploying alertmanager service with default placement... 2024-01-18T06:48:20.087 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/653e4012-b5cd-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpck5s1jz7:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcg0aieo2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2024-01-18T06:48:20.461 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.458+0000 7fa7d9759700 1 Processor -- start 2024-01-18T06:48:20.462 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.458+0000 7fa7d9759700 1 -- start start 2024-01-18T06:48:20.462 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.458+0000 7fa7d9759700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7d40fa9d0 0x7fa7d40fadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:20.462 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.458+0000 7fa7d9759700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d40fb2e0 con 0x7fa7d40fa9d0 2024-01-18T06:48:20.462 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.459+0000 7fa7d3fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7d40fa9d0 0x7fa7d40fadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:20.462 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.459+0000 7fa7d3fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7d40fa9d0 0x7fa7d40fadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60726/0 (socket says 172.21.15.47:60726) 2024-01-18T06:48:20.462 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.459+0000 7fa7d3fff700 1 -- 172.21.15.47:0/156880325 learned_addr learned my addr 172.21.15.47:0/156880325 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:48:20.462 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.459+0000 7fa7d3fff700 1 -- 172.21.15.47:0/156880325 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7d40fb420 con 0x7fa7d40fa9d0 2024-01-18T06:48:20.462 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.459+0000 7fa7d3fff700 1 --2- 172.21.15.47:0/156880325 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7d40fa9d0 0x7fa7d40fadf0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fa7c400b870 tx=0x7fa7c4006f70).ready entity=mon.0 client_cookie=167909b9eef75678 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:20.462 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.459+0000 7fa7d2ffd700 1 -- 172.21.15.47:0/156880325 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7c400da20 con 0x7fa7d40fa9d0 2024-01-18T06:48:20.462 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.459+0000 7fa7d2ffd700 1 -- 172.21.15.47:0/156880325 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa7c4010470 con 0x7fa7d40fa9d0 2024-01-18T06:48:20.462 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.459+0000 7fa7d2ffd700 1 -- 172.21.15.47:0/156880325 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7c400f5b0 con 0x7fa7d40fa9d0 2024-01-18T06:48:20.462 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.460+0000 7fa7d9759700 1 -- 172.21.15.47:0/156880325 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7d40fa9d0 msgr2=0x7fa7d40fadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:20.463 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.460+0000 7fa7d9759700 1 --2- 172.21.15.47:0/156880325 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7d40fa9d0 0x7fa7d40fadf0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fa7c400b870 tx=0x7fa7c4006f70).stop 2024-01-18T06:48:20.463 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.460+0000 7fa7d9759700 1 -- 172.21.15.47:0/156880325 shutdown_connections 2024-01-18T06:48:20.463 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.460+0000 7fa7d9759700 1 --2- 172.21.15.47:0/156880325 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7d40fa9d0 0x7fa7d40fadf0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:20.463 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.460+0000 7fa7d9759700 1 -- 172.21.15.47:0/156880325 >> 172.21.15.47:0/156880325 conn(0x7fa7d40f6210 msgr2=0x7fa7d40f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:48:20.463 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.460+0000 7fa7d9759700 1 -- 172.21.15.47:0/156880325 shutdown_connections 2024-01-18T06:48:20.463 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.460+0000 7fa7d9759700 1 -- 172.21.15.47:0/156880325 wait complete. 2024-01-18T06:48:20.463 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.460+0000 7fa7d9759700 1 Processor -- start 2024-01-18T06:48:20.463 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.461+0000 7fa7d9759700 1 -- start start 2024-01-18T06:48:20.463 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.461+0000 7fa7d9759700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7d40fa9d0 0x7fa7d4108570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:20.463 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.461+0000 7fa7d9759700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d4108a60 con 0x7fa7d40fa9d0 2024-01-18T06:48:20.463 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.461+0000 7fa7d3fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7d40fa9d0 0x7fa7d4108570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:20.463 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.461+0000 7fa7d3fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7d40fa9d0 0x7fa7d4108570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60734/0 (socket says 172.21.15.47:60734) 2024-01-18T06:48:20.463 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.461+0000 7fa7d3fff700 1 -- 172.21.15.47:0/4212575449 learned_addr learned my addr 172.21.15.47:0/4212575449 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:48:20.463 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.461+0000 7fa7d3fff700 1 -- 172.21.15.47:0/4212575449 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c400b480 con 0x7fa7d40fa9d0 2024-01-18T06:48:20.464 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.461+0000 7fa7d3fff700 1 --2- 172.21.15.47:0/4212575449 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7d40fa9d0 0x7fa7d4108570 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fa7c4006a40 tx=0x7fa7c4006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:20.465 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.461+0000 7fa7d0ff9700 1 -- 172.21.15.47:0/4212575449 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7c400f3d0 con 0x7fa7d40fa9d0 2024-01-18T06:48:20.465 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.461+0000 7fa7d0ff9700 1 -- 172.21.15.47:0/4212575449 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa7c400fef0 con 0x7fa7d40fa9d0 2024-01-18T06:48:20.465 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.461+0000 7fa7d9759700 1 -- 172.21.15.47:0/4212575449 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7d4108c60 con 0x7fa7d40fa9d0 2024-01-18T06:48:20.465 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.461+0000 7fa7d0ff9700 1 -- 172.21.15.47:0/4212575449 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7c4019d40 con 0x7fa7d40fa9d0 2024-01-18T06:48:20.465 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.462+0000 7fa7d9759700 1 -- 172.21.15.47:0/4212575449 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7d4109070 con 0x7fa7d40fa9d0 2024-01-18T06:48:20.465 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.462+0000 7fa7d0ff9700 1 -- 172.21.15.47:0/4212575449 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fa7c402b060 con 0x7fa7d40fa9d0 2024-01-18T06:48:20.465 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.462+0000 7fa7d0ff9700 1 --2- 172.21.15.47:0/4212575449 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7fa7c002feb0 0x7fa7c0032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:20.465 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.462+0000 7fa7d0ff9700 1 -- 172.21.15.47:0/4212575449 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fa7c4044230 con 0x7fa7d40fa9d0 2024-01-18T06:48:20.465 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.463+0000 7fa7d37fe700 1 --2- 172.21.15.47:0/4212575449 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7fa7c002feb0 0x7fa7c0032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:20.465 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.463+0000 7fa7d9759700 1 -- 172.21.15.47:0/4212575449 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7d4059100 con 0x7fa7d40fa9d0 2024-01-18T06:48:20.465 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.463+0000 7fa7d37fe700 1 --2- 172.21.15.47:0/4212575449 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7fa7c002feb0 0x7fa7c0032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fa7c80096a0 tx=0x7fa7c8006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:20.467 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.467+0000 7fa7d0ff9700 1 -- 172.21.15.47:0/4212575449 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa7c4014020 con 0x7fa7d40fa9d0 2024-01-18T06:48:20.655 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.654+0000 7fa7d9759700 1 -- 172.21.15.47:0/4212575449 --> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7fa7d4102490 con 0x7fa7c002feb0 2024-01-18T06:48:20.658 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.658+0000 7fa7d0ff9700 1 -- 172.21.15.47:0/4212575449 <== mgr.14142 v2:172.21.15.47:6800/1162118152 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fa7d4102490 con 0x7fa7c002feb0 2024-01-18T06:48:20.658 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-01-18T06:48:20.661 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.659+0000 7fa7d9759700 1 -- 172.21.15.47:0/4212575449 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7fa7c002feb0 msgr2=0x7fa7c0032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:20.661 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.659+0000 7fa7d9759700 1 --2- 172.21.15.47:0/4212575449 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7fa7c002feb0 0x7fa7c0032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fa7c80096a0 tx=0x7fa7c8006b40).stop 2024-01-18T06:48:20.661 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.659+0000 7fa7d9759700 1 -- 172.21.15.47:0/4212575449 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7d40fa9d0 msgr2=0x7fa7d4108570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:20.661 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.659+0000 7fa7d9759700 1 --2- 172.21.15.47:0/4212575449 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7d40fa9d0 0x7fa7d4108570 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fa7c4006a40 tx=0x7fa7c4006f70).stop 2024-01-18T06:48:20.661 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.659+0000 7fa7d9759700 1 -- 172.21.15.47:0/4212575449 shutdown_connections 2024-01-18T06:48:20.661 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.659+0000 7fa7d9759700 1 --2- 172.21.15.47:0/4212575449 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7fa7c002feb0 0x7fa7c0032370 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:20.661 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.659+0000 7fa7d9759700 1 --2- 172.21.15.47:0/4212575449 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7d40fa9d0 0x7fa7d4108570 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:20.661 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.659+0000 7fa7d9759700 1 -- 172.21.15.47:0/4212575449 >> 172.21.15.47:0/4212575449 conn(0x7fa7d40f6210 msgr2=0x7fa7d40ff760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:48:20.661 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.660+0000 7fa7d9759700 1 -- 172.21.15.47:0/4212575449 shutdown_connections 2024-01-18T06:48:20.661 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:20.660+0000 7fa7d9759700 1 -- 172.21.15.47:0/4212575449 wait complete. 2024-01-18T06:48:20.963 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:20 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:18.804560+0000 mgr.smithi047.fuhzzr (mgr.14142) 3 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-01-18T06:48:20.963 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:20 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cephadm 2024-01-18T06:48:18.805042+0000 mgr.smithi047.fuhzzr (mgr.14142) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2024-01-18T06:48:20.963 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:20 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:19.721501+0000 mgr.smithi047.fuhzzr (mgr.14142) 5 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-01-18T06:48:20.963 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:20 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cephadm 2024-01-18T06:48:19.722148+0000 mgr.smithi047.fuhzzr (mgr.14142) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2024-01-18T06:48:20.964 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:20 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:19.724283+0000 mon.smithi047 (mon.0) 79 : audit [INF] from='mgr.14142 172.21.15.47:0/122516226' entity='mgr.smithi047.fuhzzr' 2024-01-18T06:48:20.964 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:20 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:48:19.808699+0000 mon.smithi047 (mon.0) 80 : cluster [DBG] mgrmap e12: smithi047.fuhzzr(active, since 2s) 2024-01-18T06:48:20.964 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:20 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:20.659377+0000 mon.smithi047 (mon.0) 81 : audit [INF] from='mgr.14142 172.21.15.47:0/122516226' entity='mgr.smithi047.fuhzzr' 2024-01-18T06:48:21.020 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/653e4012-b5cd-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpck5s1jz7:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcg0aieo2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2024-01-18T06:48:21.514 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.511+0000 7f5455a76700 1 Processor -- start 2024-01-18T06:48:21.514 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.511+0000 7f5455a76700 1 -- start start 2024-01-18T06:48:21.514 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.511+0000 7f5455a76700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5450070430 0x7f54500fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:21.514 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.511+0000 7f5455a76700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54500fedd0 con 0x7f5450070430 2024-01-18T06:48:21.514 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.511+0000 7f544f7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5450070430 0x7f54500fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:21.514 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.511+0000 7f544f7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5450070430 0x7f54500fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60180/0 (socket says 172.21.15.47:60180) 2024-01-18T06:48:21.514 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.511+0000 7f544f7fe700 1 -- 172.21.15.47:0/174418006 learned_addr learned my addr 172.21.15.47:0/174418006 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:48:21.514 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.512+0000 7f544f7fe700 1 -- 172.21.15.47:0/174418006 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54500fef10 con 0x7f5450070430 2024-01-18T06:48:21.514 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.512+0000 7f544f7fe700 1 --2- 172.21.15.47:0/174418006 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5450070430 0x7f54500fe8e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f544000b5f0 tx=0x7f5440006ee0).ready entity=mon.0 client_cookie=96c954fe1cb2f1a2 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:21.514 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.512+0000 7f544e7fc700 1 -- 172.21.15.47:0/174418006 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f544000d600 con 0x7f5450070430 2024-01-18T06:48:21.514 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.512+0000 7f544e7fc700 1 -- 172.21.15.47:0/174418006 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f544000dbd0 con 0x7f5450070430 2024-01-18T06:48:21.514 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.512+0000 7f544e7fc700 1 -- 172.21.15.47:0/174418006 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5440011c00 con 0x7f5450070430 2024-01-18T06:48:21.514 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.512+0000 7f5455a76700 1 -- 172.21.15.47:0/174418006 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5450070430 msgr2=0x7f54500fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:21.515 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.513+0000 7f5455a76700 1 --2- 172.21.15.47:0/174418006 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5450070430 0x7f54500fe8e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f544000b5f0 tx=0x7f5440006ee0).stop 2024-01-18T06:48:21.515 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.513+0000 7f5455a76700 1 -- 172.21.15.47:0/174418006 shutdown_connections 2024-01-18T06:48:21.515 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.513+0000 7f5455a76700 1 --2- 172.21.15.47:0/174418006 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5450070430 0x7f54500fe8e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:21.515 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.513+0000 7f5455a76700 1 -- 172.21.15.47:0/174418006 >> 172.21.15.47:0/174418006 conn(0x7f54500f63c0 msgr2=0x7f54500f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:48:21.515 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.513+0000 7f5455a76700 1 -- 172.21.15.47:0/174418006 shutdown_connections 2024-01-18T06:48:21.515 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.513+0000 7f5455a76700 1 -- 172.21.15.47:0/174418006 wait complete. 2024-01-18T06:48:21.515 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.513+0000 7f5455a76700 1 Processor -- start 2024-01-18T06:48:21.515 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.513+0000 7f5455a76700 1 -- start start 2024-01-18T06:48:21.516 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.514+0000 7f5455a76700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5450070430 0x7f54500fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:21.517 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.514+0000 7f5455a76700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54500fa7e0 con 0x7f5450070430 2024-01-18T06:48:21.517 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.514+0000 7f544f7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5450070430 0x7f54500fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:21.517 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.514+0000 7f544f7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5450070430 0x7f54500fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60196/0 (socket says 172.21.15.47:60196) 2024-01-18T06:48:21.517 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.514+0000 7f544f7fe700 1 -- 172.21.15.47:0/2333484306 learned_addr learned my addr 172.21.15.47:0/2333484306 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:48:21.517 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.514+0000 7f544f7fe700 1 -- 172.21.15.47:0/2333484306 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f544000b200 con 0x7f5450070430 2024-01-18T06:48:21.517 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.514+0000 7f544f7fe700 1 --2- 172.21.15.47:0/2333484306 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5450070430 0x7f54500fc360 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f5440002f40 tx=0x7f5440006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:21.517 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.514+0000 7f5447fff700 1 -- 172.21.15.47:0/2333484306 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5440011690 con 0x7f5450070430 2024-01-18T06:48:21.517 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.514+0000 7f5455a76700 1 -- 172.21.15.47:0/2333484306 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54500fa980 con 0x7f5450070430 2024-01-18T06:48:21.517 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.514+0000 7f5447fff700 1 -- 172.21.15.47:0/2333484306 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f544001b880 con 0x7f5450070430 2024-01-18T06:48:21.517 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.514+0000 7f5447fff700 1 -- 172.21.15.47:0/2333484306 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f544001a6b0 con 0x7f5450070430 2024-01-18T06:48:21.517 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.515+0000 7f5455a76700 1 -- 172.21.15.47:0/2333484306 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54500fad90 con 0x7f5450070430 2024-01-18T06:48:21.517 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.515+0000 7f5447fff700 1 -- 172.21.15.47:0/2333484306 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f544000f070 con 0x7f5450070430 2024-01-18T06:48:21.517 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.515+0000 7f5447fff700 1 --2- 172.21.15.47:0/2333484306 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f543002fe60 0x7f5430032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:21.518 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.515+0000 7f5447fff700 1 -- 172.21.15.47:0/2333484306 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f54400421b0 con 0x7f5450070430 2024-01-18T06:48:21.518 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.515+0000 7f544effd700 1 --2- 172.21.15.47:0/2333484306 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f543002fe60 0x7f5430032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:21.518 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.516+0000 7f5455a76700 1 -- 172.21.15.47:0/2333484306 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5450059100 con 0x7f5450070430 2024-01-18T06:48:21.518 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.516+0000 7f544effd700 1 --2- 172.21.15.47:0/2333484306 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f543002fe60 0x7f5430032320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f54380096a0 tx=0x7f5438006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:21.520 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.520+0000 7f5447fff700 1 -- 172.21.15.47:0/2333484306 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5440016020 con 0x7f5450070430 2024-01-18T06:48:21.682 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.681+0000 7f5455a76700 1 -- 172.21.15.47:0/2333484306 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f54501047f0 con 0x7f5450070430 2024-01-18T06:48:21.684 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.684+0000 7f5447fff700 1 -- 172.21.15.47:0/2333484306 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v8)=0 v8) v1 ==== 142+0+0 (secure 0 0 0) 0x7f5440022900 con 0x7f5450070430 2024-01-18T06:48:21.687 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.686+0000 7f5455a76700 1 -- 172.21.15.47:0/2333484306 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f543002fe60 msgr2=0x7f5430032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:21.687 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.686+0000 7f5455a76700 1 --2- 172.21.15.47:0/2333484306 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f543002fe60 0x7f5430032320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f54380096a0 tx=0x7f5438006b40).stop 2024-01-18T06:48:21.687 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.686+0000 7f5455a76700 1 -- 172.21.15.47:0/2333484306 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5450070430 msgr2=0x7f54500fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:21.687 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.686+0000 7f5455a76700 1 --2- 172.21.15.47:0/2333484306 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5450070430 0x7f54500fc360 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f5440002f40 tx=0x7f5440006ee0).stop 2024-01-18T06:48:21.687 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.686+0000 7f5455a76700 1 -- 172.21.15.47:0/2333484306 shutdown_connections 2024-01-18T06:48:21.687 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.686+0000 7f5455a76700 1 --2- 172.21.15.47:0/2333484306 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f543002fe60 0x7f5430032320 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:21.687 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.686+0000 7f5455a76700 1 --2- 172.21.15.47:0/2333484306 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5450070430 0x7f54500fc360 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:21.687 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.686+0000 7f5455a76700 1 -- 172.21.15.47:0/2333484306 >> 172.21.15.47:0/2333484306 conn(0x7f54500f63c0 msgr2=0x7f54500f7fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:48:21.688 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.686+0000 7f5455a76700 1 -- 172.21.15.47:0/2333484306 shutdown_connections 2024-01-18T06:48:21.688 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:21.686+0000 7f5455a76700 1 -- 172.21.15.47:0/2333484306 wait complete. 2024-01-18T06:48:21.963 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:21 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:20.656709+0000 mgr.smithi047.fuhzzr (mgr.14142) 7 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-01-18T06:48:21.963 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:21 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cephadm 2024-01-18T06:48:20.657204+0000 mgr.smithi047.fuhzzr (mgr.14142) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2024-01-18T06:48:21.963 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:21 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:21.685789+0000 mon.smithi047 (mon.0) 82 : audit [INF] from='client.? 172.21.15.47:0/2333484306' entity='client.admin' 2024-01-18T06:48:22.313 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/653e4012-b5cd-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpck5s1jz7:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcg0aieo2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2024-01-18T06:48:22.735 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.734+0000 7fb218353700 1 Processor -- start 2024-01-18T06:48:22.735 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.734+0000 7fb218353700 1 -- start start 2024-01-18T06:48:22.735 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.734+0000 7fb218353700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb2100703b0 0x7fb210100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:22.735 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.734+0000 7fb218353700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2100fc750 con 0x7fb2100703b0 2024-01-18T06:48:22.735 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.734+0000 7fb2160ef700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb2100703b0 0x7fb210100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:22.736 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.734+0000 7fb2160ef700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb2100703b0 0x7fb210100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60200/0 (socket says 172.21.15.47:60200) 2024-01-18T06:48:22.736 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.734+0000 7fb2160ef700 1 -- 172.21.15.47:0/3480389383 learned_addr learned my addr 172.21.15.47:0/3480389383 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:48:22.736 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.735+0000 7fb2160ef700 1 -- 172.21.15.47:0/3480389383 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb210101250 con 0x7fb2100703b0 2024-01-18T06:48:22.740 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.735+0000 7fb2160ef700 1 --2- 172.21.15.47:0/3480389383 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb2100703b0 0x7fb210100d60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fb20c00b5f0 tx=0x7fb20c006ee0).ready entity=mon.0 client_cookie=83ae7c775a233203 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:22.740 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.735+0000 7fb2150ed700 1 -- 172.21.15.47:0/3480389383 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb20c00d600 con 0x7fb2100703b0 2024-01-18T06:48:22.740 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.735+0000 7fb2150ed700 1 -- 172.21.15.47:0/3480389383 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb20c00dc10 con 0x7fb2100703b0 2024-01-18T06:48:22.740 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.735+0000 7fb2150ed700 1 -- 172.21.15.47:0/3480389383 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb20c011cd0 con 0x7fb2100703b0 2024-01-18T06:48:22.740 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.736+0000 7fb218353700 1 -- 172.21.15.47:0/3480389383 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb2100703b0 msgr2=0x7fb210100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:22.740 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.736+0000 7fb218353700 1 --2- 172.21.15.47:0/3480389383 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb2100703b0 0x7fb210100d60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fb20c00b5f0 tx=0x7fb20c006ee0).stop 2024-01-18T06:48:22.740 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.736+0000 7fb218353700 1 -- 172.21.15.47:0/3480389383 shutdown_connections 2024-01-18T06:48:22.740 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.736+0000 7fb218353700 1 --2- 172.21.15.47:0/3480389383 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb2100703b0 0x7fb210100d60 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:22.741 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.736+0000 7fb218353700 1 -- 172.21.15.47:0/3480389383 >> 172.21.15.47:0/3480389383 conn(0x7fb2100f63c0 msgr2=0x7fb2100f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:48:22.741 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.736+0000 7fb218353700 1 -- 172.21.15.47:0/3480389383 shutdown_connections 2024-01-18T06:48:22.741 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.736+0000 7fb218353700 1 -- 172.21.15.47:0/3480389383 wait complete. 2024-01-18T06:48:22.741 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.736+0000 7fb218353700 1 Processor -- start 2024-01-18T06:48:22.741 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.736+0000 7fb218353700 1 -- start start 2024-01-18T06:48:22.741 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.737+0000 7fb218353700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb2100703b0 0x7fb21010cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:22.741 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.737+0000 7fb218353700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21010d1d0 con 0x7fb2100703b0 2024-01-18T06:48:22.741 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.737+0000 7fb2160ef700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb2100703b0 0x7fb21010cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:22.741 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.737+0000 7fb2160ef700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb2100703b0 0x7fb21010cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60212/0 (socket says 172.21.15.47:60212) 2024-01-18T06:48:22.741 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.737+0000 7fb2160ef700 1 -- 172.21.15.47:0/2523891049 learned_addr learned my addr 172.21.15.47:0/2523891049 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:48:22.741 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.737+0000 7fb2160ef700 1 -- 172.21.15.47:0/2523891049 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb20c00b200 con 0x7fb2100703b0 2024-01-18T06:48:22.741 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.737+0000 7fb2160ef700 1 --2- 172.21.15.47:0/2523891049 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb2100703b0 0x7fb21010cce0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fb20c00b1d0 tx=0x7fb20c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:22.741 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.737+0000 7fb1feffd700 1 -- 172.21.15.47:0/2523891049 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb20c011df0 con 0x7fb2100703b0 2024-01-18T06:48:22.742 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.737+0000 7fb1feffd700 1 -- 172.21.15.47:0/2523891049 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb20c01b8c0 con 0x7fb2100703b0 2024-01-18T06:48:22.742 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.737+0000 7fb218353700 1 -- 172.21.15.47:0/2523891049 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb21010d3d0 con 0x7fb2100703b0 2024-01-18T06:48:22.742 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.737+0000 7fb1feffd700 1 -- 172.21.15.47:0/2523891049 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb20c01a830 con 0x7fb2100703b0 2024-01-18T06:48:22.742 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.737+0000 7fb218353700 1 -- 172.21.15.47:0/2523891049 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb21010d7e0 con 0x7fb2100703b0 2024-01-18T06:48:22.742 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.738+0000 7fb1feffd700 1 -- 172.21.15.47:0/2523891049 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fb20c00f070 con 0x7fb2100703b0 2024-01-18T06:48:22.742 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.738+0000 7fb1feffd700 1 --2- 172.21.15.47:0/2523891049 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7fb1f802fe60 0x7fb1f8032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:22.742 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.738+0000 7fb1feffd700 1 -- 172.21.15.47:0/2523891049 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fb20c040fd0 con 0x7fb2100703b0 2024-01-18T06:48:22.742 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.738+0000 7fb218353700 1 -- 172.21.15.47:0/2523891049 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb210045ad0 con 0x7fb2100703b0 2024-01-18T06:48:22.742 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.738+0000 7fb2158ee700 1 --2- 172.21.15.47:0/2523891049 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7fb1f802fe60 0x7fb1f8032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:22.742 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.739+0000 7fb2158ee700 1 --2- 172.21.15.47:0/2523891049 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7fb1f802fe60 0x7fb1f8032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fb2000096a0 tx=0x7fb200006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:22.743 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.744+0000 7fb1feffd700 1 -- 172.21.15.47:0/2523891049 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb20c016070 con 0x7fb2100703b0 2024-01-18T06:48:22.914 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.915+0000 7fb218353700 1 -- 172.21.15.47:0/2523891049 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fb21010ed10 con 0x7fb2100703b0 2024-01-18T06:48:22.916 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.917+0000 7fb1feffd700 1 -- 172.21.15.47:0/2523891049 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v10)=0 v10) v1 ==== 130+0+0 (secure 0 0 0) 0x7fb20c019d40 con 0x7fb2100703b0 2024-01-18T06:48:22.919 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.919+0000 7fb218353700 1 -- 172.21.15.47:0/2523891049 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7fb1f802fe60 msgr2=0x7fb1f8032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:22.920 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.919+0000 7fb218353700 1 --2- 172.21.15.47:0/2523891049 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7fb1f802fe60 0x7fb1f8032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fb2000096a0 tx=0x7fb200006b40).stop 2024-01-18T06:48:22.920 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.919+0000 7fb218353700 1 -- 172.21.15.47:0/2523891049 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb2100703b0 msgr2=0x7fb21010cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:22.920 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.919+0000 7fb218353700 1 --2- 172.21.15.47:0/2523891049 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb2100703b0 0x7fb21010cce0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fb20c00b1d0 tx=0x7fb20c006ee0).stop 2024-01-18T06:48:22.920 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.919+0000 7fb218353700 1 -- 172.21.15.47:0/2523891049 shutdown_connections 2024-01-18T06:48:22.920 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.919+0000 7fb218353700 1 --2- 172.21.15.47:0/2523891049 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7fb1f802fe60 0x7fb1f8032320 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:22.920 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.919+0000 7fb218353700 1 --2- 172.21.15.47:0/2523891049 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb2100703b0 0x7fb21010cce0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:22.920 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.919+0000 7fb218353700 1 -- 172.21.15.47:0/2523891049 >> 172.21.15.47:0/2523891049 conn(0x7fb2100f63c0 msgr2=0x7fb2100f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:48:22.920 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.920+0000 7fb218353700 1 -- 172.21.15.47:0/2523891049 shutdown_connections 2024-01-18T06:48:22.920 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:22.920+0000 7fb218353700 1 -- 172.21.15.47:0/2523891049 wait complete. 2024-01-18T06:48:23.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:23 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:22.160791+0000 mon.smithi047 (mon.0) 83 : audit [INF] from='mgr.14142 172.21.15.47:0/122516226' entity='mgr.smithi047.fuhzzr' 2024-01-18T06:48:23.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:23 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:22.392331+0000 mon.smithi047 (mon.0) 84 : audit [INF] from='mgr.14142 172.21.15.47:0/122516226' entity='mgr.smithi047.fuhzzr' 2024-01-18T06:48:23.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:23 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:22.918071+0000 mon.smithi047 (mon.0) 85 : audit [INF] from='client.? 172.21.15.47:0/2523891049' entity='client.admin' 2024-01-18T06:48:23.337 INFO:teuthology.orchestra.run.smithi047.stderr:Enabling the dashboard module... 2024-01-18T06:48:23.337 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/653e4012-b5cd-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpck5s1jz7:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcg0aieo2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2024-01-18T06:48:23.793 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:23.793+0000 7f4ba4d47700 1 Processor -- start 2024-01-18T06:48:23.794 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:23.793+0000 7f4ba4d47700 1 -- start start 2024-01-18T06:48:23.794 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:23.793+0000 7f4ba4d47700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ba00fbad0 0x7f4ba00fbef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:23.794 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:23.793+0000 7f4ba4d47700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba00fc3e0 con 0x7f4ba00fbad0 2024-01-18T06:48:23.794 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:23.793+0000 7f4b9f7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ba00fbad0 0x7f4ba00fbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:23.794 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:23.793+0000 7f4b9f7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ba00fbad0 0x7f4ba00fbef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60224/0 (socket says 172.21.15.47:60224) 2024-01-18T06:48:23.794 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:23.793+0000 7f4b9f7fe700 1 -- 172.21.15.47:0/3148397818 learned_addr learned my addr 172.21.15.47:0/3148397818 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:48:23.794 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:23.793+0000 7f4b9f7fe700 1 -- 172.21.15.47:0/3148397818 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba00fc520 con 0x7f4ba00fbad0 2024-01-18T06:48:23.794 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:23.794+0000 7f4b9f7fe700 1 --2- 172.21.15.47:0/3148397818 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ba00fbad0 0x7f4ba00fbef0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f4b9000b5f0 tx=0x7f4b90006ee0).ready entity=mon.0 client_cookie=b82b7bc9fcecf07e server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:23.794 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:23.794+0000 7f4b9e7fc700 1 -- 172.21.15.47:0/3148397818 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b9000d600 con 0x7f4ba00fbad0 2024-01-18T06:48:23.794 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:23.794+0000 7f4b9e7fc700 1 -- 172.21.15.47:0/3148397818 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4b9000dc10 con 0x7f4ba00fbad0 2024-01-18T06:48:23.794 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:23.794+0000 7f4b9e7fc700 1 -- 172.21.15.47:0/3148397818 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b90011cd0 con 0x7f4ba00fbad0 2024-01-18T06:48:23.796 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:23.794+0000 7f4ba4d47700 1 -- 172.21.15.47:0/3148397818 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ba00fbad0 msgr2=0x7f4ba00fbef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:23.796 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:23.794+0000 7f4ba4d47700 1 --2- 172.21.15.47:0/3148397818 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ba00fbad0 0x7f4ba00fbef0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f4b9000b5f0 tx=0x7f4b90006ee0).stop 2024-01-18T06:48:23.796 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:23.794+0000 7f4ba4d47700 1 -- 172.21.15.47:0/3148397818 shutdown_connections 2024-01-18T06:48:23.796 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:23.794+0000 7f4ba4d47700 1 --2- 172.21.15.47:0/3148397818 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ba00fbad0 0x7f4ba00fbef0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:23.796 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:23.794+0000 7f4ba4d47700 1 -- 172.21.15.47:0/3148397818 >> 172.21.15.47:0/3148397818 conn(0x7f4ba00f7310 msgr2=0x7f4ba00f9730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:48:23.796 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:23.795+0000 7f4ba4d47700 1 -- 172.21.15.47:0/3148397818 shutdown_connections 2024-01-18T06:48:23.796 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:23.795+0000 7f4ba4d47700 1 -- 172.21.15.47:0/3148397818 wait complete. 2024-01-18T06:48:23.796 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:23.795+0000 7f4ba4d47700 1 Processor -- start 2024-01-18T06:48:23.797 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:23.795+0000 7f4ba4d47700 1 -- start start 2024-01-18T06:48:23.797 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:23.795+0000 7f4ba4d47700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ba00fbad0 0x7f4ba0072500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:23.797 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:23.795+0000 7f4ba4d47700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba00709f0 con 0x7f4ba00fbad0 2024-01-18T06:48:23.797 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:23.796+0000 7f4b9f7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ba00fbad0 0x7f4ba0072500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:23.797 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:23.796+0000 7f4b9f7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ba00fbad0 0x7f4ba0072500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60226/0 (socket says 172.21.15.47:60226) 2024-01-18T06:48:23.797 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:23.796+0000 7f4b9f7fe700 1 -- 172.21.15.47:0/2912784629 learned_addr learned my addr 172.21.15.47:0/2912784629 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:48:23.798 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:23.796+0000 7f4b9f7fe700 1 -- 172.21.15.47:0/2912784629 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b9000b200 con 0x7f4ba00fbad0 2024-01-18T06:48:23.798 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:23.796+0000 7f4b9f7fe700 1 --2- 172.21.15.47:0/2912784629 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ba00fbad0 0x7f4ba0072500 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f4b9000dfb0 tx=0x7f4b90006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:23.798 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:23.796+0000 7f4b87fff700 1 -- 172.21.15.47:0/2912784629 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b90011df0 con 0x7f4ba00fbad0 2024-01-18T06:48:23.798 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:23.796+0000 7f4b87fff700 1 -- 172.21.15.47:0/2912784629 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4b9001b8c0 con 0x7f4ba00fbad0 2024-01-18T06:48:23.798 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:23.796+0000 7f4ba4d47700 1 -- 172.21.15.47:0/2912784629 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ba0070bf0 con 0x7f4ba00fbad0 2024-01-18T06:48:23.798 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:23.796+0000 7f4b87fff700 1 -- 172.21.15.47:0/2912784629 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b9001a7c0 con 0x7f4ba00fbad0 2024-01-18T06:48:23.798 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:23.796+0000 7f4ba4d47700 1 -- 172.21.15.47:0/2912784629 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ba0071000 con 0x7f4ba00fbad0 2024-01-18T06:48:23.798 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:23.798+0000 7f4b87fff700 1 -- 172.21.15.47:0/2912784629 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f4b9000f070 con 0x7f4ba00fbad0 2024-01-18T06:48:23.798 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:23.798+0000 7f4b87fff700 1 --2- 172.21.15.47:0/2912784629 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f4b8c02fe60 0x7f4b8c032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:23.798 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:23.798+0000 7f4ba4d47700 1 -- 172.21.15.47:0/2912784629 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ba0045ad0 con 0x7f4ba00fbad0 2024-01-18T06:48:23.799 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:23.798+0000 7f4b87fff700 1 -- 172.21.15.47:0/2912784629 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f4b90041050 con 0x7f4ba00fbad0 2024-01-18T06:48:23.799 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:23.798+0000 7f4b9effd700 1 --2- 172.21.15.47:0/2912784629 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f4b8c02fe60 0x7f4b8c032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:23.799 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:23.799+0000 7f4b9effd700 1 --2- 172.21.15.47:0/2912784629 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f4b8c02fe60 0x7f4b8c032320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f4b940096a0 tx=0x7f4b94006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:23.802 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:23.803+0000 7f4b87fff700 1 -- 172.21.15.47:0/2912784629 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4b90016080 con 0x7f4ba00fbad0 2024-01-18T06:48:24.007 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:24.007+0000 7f4ba4d47700 1 -- 172.21.15.47:0/2912784629 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f4ba00fd850 con 0x7f4ba00fbad0 2024-01-18T06:48:24.713 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:24.713+0000 7f4b87fff700 1 -- 172.21.15.47:0/2912784629 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f4b9001ba30 con 0x7f4ba00fbad0 2024-01-18T06:48:24.713 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:24 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:23.712213+0000 mon.smithi047 (mon.0) 86 : audit [INF] from='mgr.14142 172.21.15.47:0/122516226' entity='mgr.smithi047.fuhzzr' 2024-01-18T06:48:24.713 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:24 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:24.008748+0000 mon.smithi047 (mon.0) 87 : audit [INF] from='client.? 172.21.15.47:0/2912784629' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-01-18T06:48:24.714 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:24.714+0000 7f4b87fff700 1 -- 172.21.15.47:0/2912784629 <== mon.0 v2:172.21.15.47:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v13) v1 ==== 88+0+0 (secure 0 0 0) 0x7f4b90023600 con 0x7f4ba00fbad0 2024-01-18T06:48:24.717 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:24.716+0000 7f4ba4d47700 1 -- 172.21.15.47:0/2912784629 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f4b8c02fe60 msgr2=0x7f4b8c032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:24.717 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:24.716+0000 7f4ba4d47700 1 --2- 172.21.15.47:0/2912784629 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f4b8c02fe60 0x7f4b8c032320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f4b940096a0 tx=0x7f4b94006b40).stop 2024-01-18T06:48:24.717 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:24.716+0000 7f4ba4d47700 1 -- 172.21.15.47:0/2912784629 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ba00fbad0 msgr2=0x7f4ba0072500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:24.717 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:24.716+0000 7f4ba4d47700 1 --2- 172.21.15.47:0/2912784629 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ba00fbad0 0x7f4ba0072500 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f4b9000dfb0 tx=0x7f4b90006ee0).stop 2024-01-18T06:48:24.717 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:24.716+0000 7f4ba4d47700 1 -- 172.21.15.47:0/2912784629 shutdown_connections 2024-01-18T06:48:24.717 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:24.716+0000 7f4ba4d47700 1 --2- 172.21.15.47:0/2912784629 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f4b8c02fe60 0x7f4b8c032320 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:24.717 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:24.716+0000 7f4ba4d47700 1 --2- 172.21.15.47:0/2912784629 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ba00fbad0 0x7f4ba0072500 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:24.717 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:24.716+0000 7f4ba4d47700 1 -- 172.21.15.47:0/2912784629 >> 172.21.15.47:0/2912784629 conn(0x7f4ba00f7310 msgr2=0x7f4ba00f8ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:48:24.717 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:24.717+0000 7f4ba4d47700 1 -- 172.21.15.47:0/2912784629 shutdown_connections 2024-01-18T06:48:24.718 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:24.717+0000 7f4ba4d47700 1 -- 172.21.15.47:0/2912784629 wait complete. 2024-01-18T06:48:25.087 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/653e4012-b5cd-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpck5s1jz7:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcg0aieo2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-01-18T06:48:25.532 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.531+0000 7f78476b2700 1 Processor -- start 2024-01-18T06:48:25.532 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.531+0000 7f78476b2700 1 -- start start 2024-01-18T06:48:25.532 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.531+0000 7f78476b2700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7840070430 0x7f78400fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:25.532 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.531+0000 7f78476b2700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78400fedd0 con 0x7f7840070430 2024-01-18T06:48:25.532 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.531+0000 7f784544e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7840070430 0x7f78400fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:25.532 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.532+0000 7f784544e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7840070430 0x7f78400fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60244/0 (socket says 172.21.15.47:60244) 2024-01-18T06:48:25.532 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.532+0000 7f784544e700 1 -- 172.21.15.47:0/3487938281 learned_addr learned my addr 172.21.15.47:0/3487938281 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:48:25.532 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.532+0000 7f784544e700 1 -- 172.21.15.47:0/3487938281 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78400fef10 con 0x7f7840070430 2024-01-18T06:48:25.532 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.532+0000 7f784544e700 1 --2- 172.21.15.47:0/3487938281 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7840070430 0x7f78400fe8e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f783000b5f0 tx=0x7f7830006ee0).ready entity=mon.0 client_cookie=3b298fdd5ea2ae83 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:25.532 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.532+0000 7f782ffff700 1 -- 172.21.15.47:0/3487938281 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f783000d600 con 0x7f7840070430 2024-01-18T06:48:25.532 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.532+0000 7f782ffff700 1 -- 172.21.15.47:0/3487938281 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f783000dc10 con 0x7f7840070430 2024-01-18T06:48:25.532 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.532+0000 7f782ffff700 1 -- 172.21.15.47:0/3487938281 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7830011cd0 con 0x7f7840070430 2024-01-18T06:48:25.534 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.533+0000 7f78476b2700 1 -- 172.21.15.47:0/3487938281 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7840070430 msgr2=0x7f78400fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:25.534 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.533+0000 7f78476b2700 1 --2- 172.21.15.47:0/3487938281 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7840070430 0x7f78400fe8e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f783000b5f0 tx=0x7f7830006ee0).stop 2024-01-18T06:48:25.534 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.533+0000 7f78476b2700 1 -- 172.21.15.47:0/3487938281 shutdown_connections 2024-01-18T06:48:25.534 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.533+0000 7f78476b2700 1 --2- 172.21.15.47:0/3487938281 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7840070430 0x7f78400fe8e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:25.534 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.533+0000 7f78476b2700 1 -- 172.21.15.47:0/3487938281 >> 172.21.15.47:0/3487938281 conn(0x7f78400f63c0 msgr2=0x7f78400f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:48:25.534 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.533+0000 7f78476b2700 1 -- 172.21.15.47:0/3487938281 shutdown_connections 2024-01-18T06:48:25.534 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.533+0000 7f78476b2700 1 -- 172.21.15.47:0/3487938281 wait complete. 2024-01-18T06:48:25.534 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.534+0000 7f78476b2700 1 Processor -- start 2024-01-18T06:48:25.534 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.534+0000 7f78476b2700 1 -- start start 2024-01-18T06:48:25.535 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.534+0000 7f78476b2700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7840070430 0x7f784010ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:25.535 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.534+0000 7f78476b2700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f784010aff0 con 0x7f7840070430 2024-01-18T06:48:25.535 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.534+0000 7f784544e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7840070430 0x7f784010ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:25.535 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.534+0000 7f784544e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7840070430 0x7f784010ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60256/0 (socket says 172.21.15.47:60256) 2024-01-18T06:48:25.535 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.534+0000 7f784544e700 1 -- 172.21.15.47:0/2791163524 learned_addr learned my addr 172.21.15.47:0/2791163524 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:48:25.535 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.534+0000 7f784544e700 1 -- 172.21.15.47:0/2791163524 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f783000b200 con 0x7f7840070430 2024-01-18T06:48:25.536 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.535+0000 7f784544e700 1 --2- 172.21.15.47:0/2791163524 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7840070430 0x7f784010ab00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f7830006800 tx=0x7f7830006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:25.536 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.535+0000 7f782dffb700 1 -- 172.21.15.47:0/2791163524 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7830011df0 con 0x7f7840070430 2024-01-18T06:48:25.536 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.535+0000 7f782dffb700 1 -- 172.21.15.47:0/2791163524 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f783001b8c0 con 0x7f7840070430 2024-01-18T06:48:25.536 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.535+0000 7f78476b2700 1 -- 172.21.15.47:0/2791163524 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f784010b1f0 con 0x7f7840070430 2024-01-18T06:48:25.536 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.535+0000 7f782dffb700 1 -- 172.21.15.47:0/2791163524 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f783001a830 con 0x7f7840070430 2024-01-18T06:48:25.536 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.535+0000 7f78476b2700 1 -- 172.21.15.47:0/2791163524 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f784010b600 con 0x7f7840070430 2024-01-18T06:48:25.536 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.536+0000 7f782dffb700 1 -- 172.21.15.47:0/2791163524 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f783000f070 con 0x7f7840070430 2024-01-18T06:48:25.536 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.536+0000 7f782dffb700 1 --2- 172.21.15.47:0/2791163524 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f782802ff00 0x7f78280323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:25.536 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.536+0000 7f782dffb700 1 -- 172.21.15.47:0/2791163524 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f7830041ef0 con 0x7f7840070430 2024-01-18T06:48:25.536 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.536+0000 7f7844c4d700 1 -- 172.21.15.47:0/2791163524 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f782802ff00 msgr2=0x7f78280323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/1162118152 2024-01-18T06:48:25.537 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.536+0000 7f7844c4d700 1 --2- 172.21.15.47:0/2791163524 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f782802ff00 0x7f78280323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-18T06:48:25.537 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.536+0000 7f78476b2700 1 -- 172.21.15.47:0/2791163524 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7840059100 con 0x7f7840070430 2024-01-18T06:48:25.543 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.544+0000 7f782dffb700 1 -- 172.21.15.47:0/2791163524 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7830024410 con 0x7f7840070430 2024-01-18T06:48:25.736 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.737+0000 7f7844c4d700 1 -- 172.21.15.47:0/2791163524 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f782802ff00 msgr2=0x7f78280323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/1162118152 2024-01-18T06:48:25.736 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.737+0000 7f7844c4d700 1 --2- 172.21.15.47:0/2791163524 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f782802ff00 0x7f78280323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-18T06:48:25.742 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.743+0000 7f78476b2700 1 -- 172.21.15.47:0/2791163524 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f7840104860 con 0x7f7840070430 2024-01-18T06:48:25.743 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.743+0000 7f782dffb700 1 -- 172.21.15.47:0/2791163524 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v13) v1 ==== 56+0+104 (secure 0 0 0) 0x7f78300107d0 con 0x7f7840070430 2024-01-18T06:48:25.743 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: { 2024-01-18T06:48:25.743 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "epoch": 13, 2024-01-18T06:48:25.743 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "available": true, 2024-01-18T06:48:25.743 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "active_name": "smithi047.fuhzzr", 2024-01-18T06:48:25.743 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "num_standby": 0 2024-01-18T06:48:25.744 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: } 2024-01-18T06:48:25.745 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.745+0000 7f78476b2700 1 -- 172.21.15.47:0/2791163524 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f782802ff00 msgr2=0x7f78280323c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-18T06:48:25.745 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.745+0000 7f78476b2700 1 --2- 172.21.15.47:0/2791163524 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f782802ff00 0x7f78280323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-18T06:48:25.745 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.745+0000 7f78476b2700 1 -- 172.21.15.47:0/2791163524 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7840070430 msgr2=0x7f784010ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:25.745 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.745+0000 7f78476b2700 1 --2- 172.21.15.47:0/2791163524 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7840070430 0x7f784010ab00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f7830006800 tx=0x7f7830006ee0).stop 2024-01-18T06:48:25.746 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.745+0000 7f78476b2700 1 -- 172.21.15.47:0/2791163524 shutdown_connections 2024-01-18T06:48:25.746 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.745+0000 7f78476b2700 1 --2- 172.21.15.47:0/2791163524 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f782802ff00 0x7f78280323c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-18T06:48:25.746 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.745+0000 7f78476b2700 1 --2- 172.21.15.47:0/2791163524 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7840070430 0x7f784010ab00 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:25.746 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.745+0000 7f78476b2700 1 -- 172.21.15.47:0/2791163524 >> 172.21.15.47:0/2791163524 conn(0x7f78400f63c0 msgr2=0x7f78400f7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:48:25.746 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.745+0000 7f78476b2700 1 -- 172.21.15.47:0/2791163524 shutdown_connections 2024-01-18T06:48:25.746 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:25.745+0000 7f78476b2700 1 -- 172.21.15.47:0/2791163524 wait complete. 2024-01-18T06:48:25.963 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:25 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:24.715172+0000 mon.smithi047 (mon.0) 88 : audit [INF] from='client.? 172.21.15.47:0/2912784629' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-01-18T06:48:25.963 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:25 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:48:24.715220+0000 mon.smithi047 (mon.0) 89 : cluster [DBG] mgrmap e13: smithi047.fuhzzr(active, since 7s) 2024-01-18T06:48:26.120 INFO:teuthology.orchestra.run.smithi047.stderr:Waiting for the mgr to restart... 2024-01-18T06:48:26.120 INFO:teuthology.orchestra.run.smithi047.stderr:Waiting for mgr epoch 13... 2024-01-18T06:48:26.121 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/653e4012-b5cd-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpck5s1jz7:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcg0aieo2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-01-18T06:48:26.461 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:26.459+0000 7f89d1090700 1 Processor -- start 2024-01-18T06:48:26.461 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:26.459+0000 7f89d1090700 1 -- start start 2024-01-18T06:48:26.461 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:26.460+0000 7f89d1090700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89cc0703d0 0x7f89cc100d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:26.462 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:26.460+0000 7f89d1090700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89cc0fc730 con 0x7f89cc0703d0 2024-01-18T06:48:26.462 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:26.460+0000 7f89cad9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89cc0703d0 0x7f89cc100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:26.462 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:26.460+0000 7f89cad9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89cc0703d0 0x7f89cc100d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60264/0 (socket says 172.21.15.47:60264) 2024-01-18T06:48:26.462 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:26.460+0000 7f89cad9d700 1 -- 172.21.15.47:0/2958943089 learned_addr learned my addr 172.21.15.47:0/2958943089 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:48:26.462 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:26.460+0000 7f89cad9d700 1 -- 172.21.15.47:0/2958943089 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89cc101230 con 0x7f89cc0703d0 2024-01-18T06:48:26.462 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:26.460+0000 7f89cad9d700 1 --2- 172.21.15.47:0/2958943089 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89cc0703d0 0x7f89cc100d40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f89bc00b5f0 tx=0x7f89bc006ee0).ready entity=mon.0 client_cookie=2955e5a1e756f487 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:26.462 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:26.460+0000 7f89c9d9b700 1 -- 172.21.15.47:0/2958943089 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89bc00d600 con 0x7f89cc0703d0 2024-01-18T06:48:26.462 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:26.460+0000 7f89c9d9b700 1 -- 172.21.15.47:0/2958943089 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f89bc00dc10 con 0x7f89cc0703d0 2024-01-18T06:48:26.462 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:26.460+0000 7f89c9d9b700 1 -- 172.21.15.47:0/2958943089 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89bc011cd0 con 0x7f89cc0703d0 2024-01-18T06:48:26.462 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:26.461+0000 7f89d1090700 1 -- 172.21.15.47:0/2958943089 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89cc0703d0 msgr2=0x7f89cc100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:26.462 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:26.461+0000 7f89d1090700 1 --2- 172.21.15.47:0/2958943089 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89cc0703d0 0x7f89cc100d40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f89bc00b5f0 tx=0x7f89bc006ee0).stop 2024-01-18T06:48:26.462 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:26.461+0000 7f89d1090700 1 -- 172.21.15.47:0/2958943089 shutdown_connections 2024-01-18T06:48:26.462 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:26.461+0000 7f89d1090700 1 --2- 172.21.15.47:0/2958943089 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89cc0703d0 0x7f89cc100d40 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:26.462 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:26.461+0000 7f89d1090700 1 -- 172.21.15.47:0/2958943089 >> 172.21.15.47:0/2958943089 conn(0x7f89cc0f63c0 msgr2=0x7f89cc0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:48:26.463 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:26.461+0000 7f89d1090700 1 -- 172.21.15.47:0/2958943089 shutdown_connections 2024-01-18T06:48:26.463 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:26.461+0000 7f89d1090700 1 -- 172.21.15.47:0/2958943089 wait complete. 2024-01-18T06:48:26.463 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:26.462+0000 7f89d1090700 1 Processor -- start 2024-01-18T06:48:26.463 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:26.462+0000 7f89d1090700 1 -- start start 2024-01-18T06:48:26.463 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:26.462+0000 7f89d1090700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89cc0703d0 0x7f89cc10cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:26.463 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:26.462+0000 7f89d1090700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89cc10d1f0 con 0x7f89cc0703d0 2024-01-18T06:48:26.465 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:26.462+0000 7f89cad9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89cc0703d0 0x7f89cc10cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:26.465 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:26.462+0000 7f89cad9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89cc0703d0 0x7f89cc10cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:60272/0 (socket says 172.21.15.47:60272) 2024-01-18T06:48:26.465 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:26.462+0000 7f89cad9d700 1 -- 172.21.15.47:0/3932847068 learned_addr learned my addr 172.21.15.47:0/3932847068 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:48:26.465 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:26.463+0000 7f89cad9d700 1 -- 172.21.15.47:0/3932847068 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89bc00b200 con 0x7f89cc0703d0 2024-01-18T06:48:26.465 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:26.463+0000 7f89cad9d700 1 --2- 172.21.15.47:0/3932847068 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89cc0703d0 0x7f89cc10cd00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f89bc006a40 tx=0x7f89bc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:26.465 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:26.463+0000 7f89b37fe700 1 -- 172.21.15.47:0/3932847068 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89bc011c90 con 0x7f89cc0703d0 2024-01-18T06:48:26.465 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:26.463+0000 7f89b37fe700 1 -- 172.21.15.47:0/3932847068 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f89bc01b8c0 con 0x7f89cc0703d0 2024-01-18T06:48:26.465 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:26.463+0000 7f89d1090700 1 -- 172.21.15.47:0/3932847068 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89cc10d3f0 con 0x7f89cc0703d0 2024-01-18T06:48:26.465 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:26.463+0000 7f89b37fe700 1 -- 172.21.15.47:0/3932847068 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89bc01a8c0 con 0x7f89cc0703d0 2024-01-18T06:48:26.465 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:26.463+0000 7f89d1090700 1 -- 172.21.15.47:0/3932847068 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89cc10d800 con 0x7f89cc0703d0 2024-01-18T06:48:26.465 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:26.464+0000 7f89b37fe700 1 -- 172.21.15.47:0/3932847068 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f89bc00f070 con 0x7f89cc0703d0 2024-01-18T06:48:26.465 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:26.464+0000 7f89b37fe700 1 --2- 172.21.15.47:0/3932847068 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f89b802feb0 0x7f89b8032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:26.465 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:26.464+0000 7f89d1090700 1 -- 172.21.15.47:0/3932847068 --> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f89cc059100 con 0x7f89b802feb0 2024-01-18T06:48:26.466 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:26.464+0000 7f89b37fe700 1 -- 172.21.15.47:0/3932847068 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f89bc040a70 con 0x7f89cc0703d0 2024-01-18T06:48:26.466 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:26.464+0000 7f89ca59c700 1 -- 172.21.15.47:0/3932847068 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f89b802feb0 msgr2=0x7f89b8032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/1162118152 2024-01-18T06:48:26.466 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:26.464+0000 7f89ca59c700 1 --2- 172.21.15.47:0/3932847068 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f89b802feb0 0x7f89b8032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-18T06:48:26.664 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:26.665+0000 7f89ca59c700 1 -- 172.21.15.47:0/3932847068 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f89b802feb0 msgr2=0x7f89b8032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/1162118152 2024-01-18T06:48:26.664 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:26.665+0000 7f89ca59c700 1 --2- 172.21.15.47:0/3932847068 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f89b802feb0 0x7f89b8032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-18T06:48:26.963 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:26 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:25.744277+0000 mon.smithi047 (mon.0) 90 : audit [DBG] from='client.? 172.21.15.47:0/2791163524' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-18T06:48:27.065 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:27.066+0000 7f89ca59c700 1 -- 172.21.15.47:0/3932847068 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f89b802feb0 msgr2=0x7f89b8032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/1162118152 2024-01-18T06:48:27.065 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:27.066+0000 7f89ca59c700 1 --2- 172.21.15.47:0/3932847068 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f89b802feb0 0x7f89b8032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-01-18T06:48:27.867 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:27.867+0000 7f89ca59c700 1 -- 172.21.15.47:0/3932847068 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f89b802feb0 msgr2=0x7f89b8032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/1162118152 2024-01-18T06:48:27.867 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:27.867+0000 7f89ca59c700 1 --2- 172.21.15.47:0/3932847068 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f89b802feb0 0x7f89b8032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-01-18T06:48:29.468 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:29.468+0000 7f89ca59c700 1 -- 172.21.15.47:0/3932847068 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f89b802feb0 msgr2=0x7f89b8032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.47:6800/1162118152 2024-01-18T06:48:29.468 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:29.468+0000 7f89ca59c700 1 --2- 172.21.15.47:0/3932847068 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f89b802feb0 0x7f89b8032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-01-18T06:48:29.969 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:29.968+0000 7f89b37fe700 1 -- 172.21.15.47:0/3932847068 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mgrmap(e 14) v1 ==== 35997+0+0 (secure 0 0 0) 0x7f89bc0245a0 con 0x7f89cc0703d0 2024-01-18T06:48:29.969 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:29.968+0000 7f89b37fe700 1 -- 172.21.15.47:0/3932847068 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f89b802feb0 msgr2=0x7f89b8032370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-18T06:48:29.969 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:29.968+0000 7f89b37fe700 1 --2- 172.21.15.47:0/3932847068 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f89b802feb0 0x7f89b8032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-18T06:48:30.111 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:29 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:48:29.916471+0000 mon.smithi047 (mon.0) 91 : cluster [INF] Active manager daemon smithi047.fuhzzr restarted 2024-01-18T06:48:30.111 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:29 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:48:29.916868+0000 mon.smithi047 (mon.0) 92 : cluster [INF] Activating manager daemon smithi047.fuhzzr 2024-01-18T06:48:30.111 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:29 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:48:29.918465+0000 mon.smithi047 (mon.0) 93 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-01-18T06:48:30.972 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:30.971+0000 7f89b37fe700 1 -- 172.21.15.47:0/3932847068 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f89bc0405b0 con 0x7f89cc0703d0 2024-01-18T06:48:30.972 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:30.971+0000 7f89b37fe700 1 --2- 172.21.15.47:0/3932847068 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f89b8033990 0x7f89b8035d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:30.972 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:30.971+0000 7f89b37fe700 1 -- 172.21.15.47:0/3932847068 --> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f89cc059100 con 0x7f89b8033990 2024-01-18T06:48:30.972 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:30.972+0000 7f89ca59c700 1 --2- 172.21.15.47:0/3932847068 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f89b8033990 0x7f89b8035d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:30.972 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:30.972+0000 7f89ca59c700 1 --2- 172.21.15.47:0/3932847068 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f89b8033990 0x7f89b8035d80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f89c0006fd0 tx=0x7f89c0006cf0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:30.973 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:30.973+0000 7f89b37fe700 1 -- 172.21.15.47:0/3932847068 <== mgr.14162 v2:172.21.15.47:6800/1933549322 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f89cc059100 con 0x7f89b8033990 2024-01-18T06:48:30.979 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:30.980+0000 7f89d1090700 1 -- 172.21.15.47:0/3932847068 --> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f89cc045ad0 con 0x7f89b8033990 2024-01-18T06:48:30.980 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:30.980+0000 7f89b37fe700 1 -- 172.21.15.47:0/3932847068 <== mgr.14162 v2:172.21.15.47:6800/1933549322 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f89cc045ad0 con 0x7f89b8033990 2024-01-18T06:48:30.980 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: { 2024-01-18T06:48:30.980 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "mgrmap_epoch": 15, 2024-01-18T06:48:30.981 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: "initialized": true 2024-01-18T06:48:30.981 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: } 2024-01-18T06:48:30.981 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:30.980+0000 7f89d1090700 1 -- 172.21.15.47:0/3932847068 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f89b8033990 msgr2=0x7f89b8035d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:30.981 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:30.980+0000 7f89d1090700 1 --2- 172.21.15.47:0/3932847068 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f89b8033990 0x7f89b8035d80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f89c0006fd0 tx=0x7f89c0006cf0).stop 2024-01-18T06:48:30.981 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:30.981+0000 7f89d1090700 1 -- 172.21.15.47:0/3932847068 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89cc0703d0 msgr2=0x7f89cc10cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:30.981 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:30.981+0000 7f89d1090700 1 --2- 172.21.15.47:0/3932847068 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89cc0703d0 0x7f89cc10cd00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f89bc006a40 tx=0x7f89bc006ee0).stop 2024-01-18T06:48:30.981 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:30.981+0000 7f89d1090700 1 -- 172.21.15.47:0/3932847068 shutdown_connections 2024-01-18T06:48:30.981 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:30.981+0000 7f89d1090700 1 --2- 172.21.15.47:0/3932847068 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f89b8033990 0x7f89b8035d80 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:30.981 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:30.981+0000 7f89d1090700 1 --2- 172.21.15.47:0/3932847068 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89cc0703d0 0x7f89cc10cd00 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:30.981 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:30.981+0000 7f89d1090700 1 --2- 172.21.15.47:0/3932847068 >> [v2:172.21.15.47:6800/1162118152,v1:172.21.15.47:6801/1162118152] conn(0x7f89b802feb0 0x7f89b8032370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-18T06:48:30.981 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:30.981+0000 7f89d1090700 1 -- 172.21.15.47:0/3932847068 >> 172.21.15.47:0/3932847068 conn(0x7f89cc0f63c0 msgr2=0x7f89cc0f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:48:30.982 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:30.981+0000 7f89d1090700 1 -- 172.21.15.47:0/3932847068 shutdown_connections 2024-01-18T06:48:30.982 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:30.981+0000 7f89d1090700 1 -- 172.21.15.47:0/3932847068 wait complete. 2024-01-18T06:48:31.152 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:30 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:48:29.969237+0000 mon.smithi047 (mon.0) 94 : cluster [DBG] mgrmap e14: smithi047.fuhzzr(active, starting, since 0.052452s) 2024-01-18T06:48:31.152 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:30 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:29.971739+0000 mon.smithi047 (mon.0) 95 : audit [DBG] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix": "mon metadata", "id": "smithi047"}]: dispatch 2024-01-18T06:48:31.152 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:30 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:29.972502+0000 mon.smithi047 (mon.0) 96 : audit [DBG] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix": "mgr metadata", "who": "smithi047.fuhzzr", "id": "smithi047.fuhzzr"}]: dispatch 2024-01-18T06:48:31.152 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:30 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:29.973124+0000 mon.smithi047 (mon.0) 97 : audit [DBG] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-18T06:48:31.152 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:30 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:29.973355+0000 mon.smithi047 (mon.0) 98 : audit [DBG] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-18T06:48:31.152 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:30 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:29.973595+0000 mon.smithi047 (mon.0) 99 : audit [DBG] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-18T06:48:31.152 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:30 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:48:29.986895+0000 mon.smithi047 (mon.0) 100 : cluster [INF] Manager daemon smithi047.fuhzzr is now available 2024-01-18T06:48:31.152 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:30 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:30.090584+0000 mon.smithi047 (mon.0) 101 : audit [DBG] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-18T06:48:31.152 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:30 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:30.093306+0000 mon.smithi047 (mon.0) 102 : audit [DBG] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-18T06:48:31.153 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:30 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:30.108999+0000 mon.smithi047 (mon.0) 103 : audit [INF] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.fuhzzr/mirror_snapshot_schedule"}]: dispatch 2024-01-18T06:48:31.153 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:30 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:30.112406+0000 mon.smithi047 (mon.0) 104 : audit [INF] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.fuhzzr/trash_purge_schedule"}]: dispatch 2024-01-18T06:48:31.336 INFO:teuthology.orchestra.run.smithi047.stderr:mgr epoch 13 is available 2024-01-18T06:48:31.336 INFO:teuthology.orchestra.run.smithi047.stderr:Generating a dashboard self-signed certificate... 2024-01-18T06:48:31.337 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/653e4012-b5cd-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpck5s1jz7:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcg0aieo2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2024-01-18T06:48:31.797 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:31.796+0000 7f730a1a7700 1 Processor -- start 2024-01-18T06:48:31.797 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:31.796+0000 7f730a1a7700 1 -- start start 2024-01-18T06:48:31.797 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:31.796+0000 7f730a1a7700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73040fa080 0x7f73040fa4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:31.797 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:31.796+0000 7f730a1a7700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73040fa990 con 0x7f73040fa080 2024-01-18T06:48:31.797 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:31.797+0000 7f73037fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73040fa080 0x7f73040fa4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:31.797 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:31.797+0000 7f73037fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73040fa080 0x7f73040fa4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36216/0 (socket says 172.21.15.47:36216) 2024-01-18T06:48:31.797 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:31.797+0000 7f73037fe700 1 -- 172.21.15.47:0/3323132650 learned_addr learned my addr 172.21.15.47:0/3323132650 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:48:31.797 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:31.797+0000 7f73037fe700 1 -- 172.21.15.47:0/3323132650 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73040faad0 con 0x7f73040fa080 2024-01-18T06:48:31.797 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:31.797+0000 7f73037fe700 1 --2- 172.21.15.47:0/3323132650 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73040fa080 0x7f73040fa4a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f72f800b5f0 tx=0x7f72f8006ee0).ready entity=mon.0 client_cookie=ee283f53f00e10b5 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:31.798 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:31.797+0000 7f73027fc700 1 -- 172.21.15.47:0/3323132650 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72f800d600 con 0x7f73040fa080 2024-01-18T06:48:31.798 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:31.797+0000 7f73027fc700 1 -- 172.21.15.47:0/3323132650 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f72f800dc10 con 0x7f73040fa080 2024-01-18T06:48:31.798 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:31.797+0000 7f73027fc700 1 -- 172.21.15.47:0/3323132650 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72f8011cd0 con 0x7f73040fa080 2024-01-18T06:48:31.799 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:31.798+0000 7f730a1a7700 1 -- 172.21.15.47:0/3323132650 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73040fa080 msgr2=0x7f73040fa4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:31.799 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:31.798+0000 7f730a1a7700 1 --2- 172.21.15.47:0/3323132650 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73040fa080 0x7f73040fa4a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f72f800b5f0 tx=0x7f72f8006ee0).stop 2024-01-18T06:48:31.799 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:31.798+0000 7f730a1a7700 1 -- 172.21.15.47:0/3323132650 shutdown_connections 2024-01-18T06:48:31.799 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:31.798+0000 7f730a1a7700 1 --2- 172.21.15.47:0/3323132650 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73040fa080 0x7f73040fa4a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:31.800 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:31.798+0000 7f730a1a7700 1 -- 172.21.15.47:0/3323132650 >> 172.21.15.47:0/3323132650 conn(0x7f73040f5b40 msgr2=0x7f73040f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:48:31.800 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:31.798+0000 7f730a1a7700 1 -- 172.21.15.47:0/3323132650 shutdown_connections 2024-01-18T06:48:31.800 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:31.798+0000 7f730a1a7700 1 -- 172.21.15.47:0/3323132650 wait complete. 2024-01-18T06:48:31.800 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:31.799+0000 7f730a1a7700 1 Processor -- start 2024-01-18T06:48:31.800 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:31.799+0000 7f730a1a7700 1 -- start start 2024-01-18T06:48:31.800 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:31.799+0000 7f730a1a7700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73040fa080 0x7f730410ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:31.800 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:31.799+0000 7f730a1a7700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f730410b1d0 con 0x7f73040fa080 2024-01-18T06:48:31.800 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:31.799+0000 7f73037fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73040fa080 0x7f730410ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:31.800 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:31.799+0000 7f73037fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73040fa080 0x7f730410ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36226/0 (socket says 172.21.15.47:36226) 2024-01-18T06:48:31.800 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:31.799+0000 7f73037fe700 1 -- 172.21.15.47:0/785989223 learned_addr learned my addr 172.21.15.47:0/785989223 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:48:31.801 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:31.799+0000 7f73037fe700 1 -- 172.21.15.47:0/785989223 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f800b200 con 0x7f73040fa080 2024-01-18T06:48:31.801 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:31.800+0000 7f73037fe700 1 --2- 172.21.15.47:0/785989223 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73040fa080 0x7f730410ace0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f72f800e040 tx=0x7f72f8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:31.801 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:31.800+0000 7f72ebfff700 1 -- 172.21.15.47:0/785989223 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72f801b410 con 0x7f73040fa080 2024-01-18T06:48:31.801 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:31.800+0000 7f730a1a7700 1 -- 172.21.15.47:0/785989223 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f730410b3d0 con 0x7f73040fa080 2024-01-18T06:48:31.801 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:31.800+0000 7f72ebfff700 1 -- 172.21.15.47:0/785989223 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f72f801ba20 con 0x7f73040fa080 2024-01-18T06:48:31.802 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:31.800+0000 7f72ebfff700 1 -- 172.21.15.47:0/785989223 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72f801a910 con 0x7f73040fa080 2024-01-18T06:48:31.802 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:31.800+0000 7f730a1a7700 1 -- 172.21.15.47:0/785989223 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f730410b7e0 con 0x7f73040fa080 2024-01-18T06:48:31.802 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:31.801+0000 7f72ebfff700 1 -- 172.21.15.47:0/785989223 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f72f800f070 con 0x7f73040fa080 2024-01-18T06:48:31.802 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:31.801+0000 7f730a1a7700 1 -- 172.21.15.47:0/785989223 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7304045ad0 con 0x7f73040fa080 2024-01-18T06:48:31.802 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:31.801+0000 7f72ebfff700 1 --2- 172.21.15.47:0/785989223 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f72f002fd50 0x7f72f0032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:31.802 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:31.801+0000 7f72ebfff700 1 -- 172.21.15.47:0/785989223 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f72f80411b0 con 0x7f73040fa080 2024-01-18T06:48:31.802 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:31.801+0000 7f7302ffd700 1 --2- 172.21.15.47:0/785989223 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f72f002fd50 0x7f72f0032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:31.802 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:31.801+0000 7f7302ffd700 1 --2- 172.21.15.47:0/785989223 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f72f002fd50 0x7f72f0032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f72f40096a0 tx=0x7f72f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:31.804 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:31.805+0000 7f72ebfff700 1 -- 172.21.15.47:0/785989223 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f72f8010be0 con 0x7f73040fa080 2024-01-18T06:48:31.971 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:31.972+0000 7f730a1a7700 1 -- 172.21.15.47:0/785989223 --> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f7304104c10 con 0x7f72f002fd50 2024-01-18T06:48:32.023 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:32.024+0000 7f72ebfff700 1 -- 172.21.15.47:0/785989223 <== mgr.14162 v2:172.21.15.47:6800/1933549322 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f7304104c10 con 0x7f72f002fd50 2024-01-18T06:48:32.024 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: Self-signed certificate created 2024-01-18T06:48:32.026 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:32.026+0000 7f730a1a7700 1 -- 172.21.15.47:0/785989223 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f72f002fd50 msgr2=0x7f72f0032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:32.026 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:32.026+0000 7f730a1a7700 1 --2- 172.21.15.47:0/785989223 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f72f002fd50 0x7f72f0032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f72f40096a0 tx=0x7f72f4006b40).stop 2024-01-18T06:48:32.026 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:32.026+0000 7f730a1a7700 1 -- 172.21.15.47:0/785989223 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73040fa080 msgr2=0x7f730410ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:32.026 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:32.026+0000 7f730a1a7700 1 --2- 172.21.15.47:0/785989223 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73040fa080 0x7f730410ace0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f72f800e040 tx=0x7f72f8006ee0).stop 2024-01-18T06:48:32.026 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:32.026+0000 7f730a1a7700 1 -- 172.21.15.47:0/785989223 shutdown_connections 2024-01-18T06:48:32.026 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:32.026+0000 7f730a1a7700 1 --2- 172.21.15.47:0/785989223 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f72f002fd50 0x7f72f0032210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:32.027 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:32.026+0000 7f730a1a7700 1 --2- 172.21.15.47:0/785989223 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73040fa080 0x7f730410ace0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:32.027 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:32.026+0000 7f730a1a7700 1 -- 172.21.15.47:0/785989223 >> 172.21.15.47:0/785989223 conn(0x7f73040f5b40 msgr2=0x7f73040f7570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:48:32.027 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:32.026+0000 7f730a1a7700 1 -- 172.21.15.47:0/785989223 shutdown_connections 2024-01-18T06:48:32.027 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:32.026+0000 7f730a1a7700 1 -- 172.21.15.47:0/785989223 wait complete. 2024-01-18T06:48:32.201 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:31 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:30.973733+0000 mgr.smithi047.fuhzzr (mgr.14162) 1 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-18T06:48:32.201 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:31 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:48:30.974356+0000 mon.smithi047 (mon.0) 105 : cluster [DBG] mgrmap e15: smithi047.fuhzzr(active, since 1.05757s) 2024-01-18T06:48:32.201 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:31 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:30.981003+0000 mgr.smithi047.fuhzzr (mgr.14162) 2 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-18T06:48:32.201 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:31 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:31.030860+0000 mon.smithi047 (mon.0) 106 : audit [INF] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' 2024-01-18T06:48:32.703 INFO:teuthology.orchestra.run.smithi047.stderr:Creating initial admin user... 2024-01-18T06:48:32.704 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/653e4012-b5cd-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpck5s1jz7:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcg0aieo2:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp54fqdtif:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2024-01-18T06:48:33.076 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.071+0000 7f62fd20d700 1 Processor -- start 2024-01-18T06:48:33.076 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.071+0000 7f62fd20d700 1 -- start start 2024-01-18T06:48:33.076 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.072+0000 7f62fd20d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62f8070390 0x7f62f8100d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:33.076 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.072+0000 7f62fd20d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f80fc770 con 0x7f62f8070390 2024-01-18T06:48:33.076 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.072+0000 7f62f6d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62f8070390 0x7f62f8100d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:33.076 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.072+0000 7f62f6d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62f8070390 0x7f62f8100d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36236/0 (socket says 172.21.15.47:36236) 2024-01-18T06:48:33.077 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.072+0000 7f62f6d9d700 1 -- 172.21.15.47:0/1851432164 learned_addr learned my addr 172.21.15.47:0/1851432164 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:48:33.077 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.072+0000 7f62f6d9d700 1 -- 172.21.15.47:0/1851432164 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f8101270 con 0x7f62f8070390 2024-01-18T06:48:33.077 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.072+0000 7f62f6d9d700 1 --2- 172.21.15.47:0/1851432164 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62f8070390 0x7f62f8100d80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f62ec00b5f0 tx=0x7f62ec006ee0).ready entity=mon.0 client_cookie=f7eef66d7331149b server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:33.077 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.072+0000 7f62f5d9b700 1 -- 172.21.15.47:0/1851432164 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62ec00d600 con 0x7f62f8070390 2024-01-18T06:48:33.077 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.072+0000 7f62f5d9b700 1 -- 172.21.15.47:0/1851432164 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f62ec00dc10 con 0x7f62f8070390 2024-01-18T06:48:33.077 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.072+0000 7f62f5d9b700 1 -- 172.21.15.47:0/1851432164 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62ec011cd0 con 0x7f62f8070390 2024-01-18T06:48:33.078 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.073+0000 7f62fd20d700 1 -- 172.21.15.47:0/1851432164 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62f8070390 msgr2=0x7f62f8100d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:33.078 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.073+0000 7f62fd20d700 1 --2- 172.21.15.47:0/1851432164 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62f8070390 0x7f62f8100d80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f62ec00b5f0 tx=0x7f62ec006ee0).stop 2024-01-18T06:48:33.078 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.073+0000 7f62fd20d700 1 -- 172.21.15.47:0/1851432164 shutdown_connections 2024-01-18T06:48:33.078 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.073+0000 7f62fd20d700 1 --2- 172.21.15.47:0/1851432164 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62f8070390 0x7f62f8100d80 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:33.078 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.073+0000 7f62fd20d700 1 -- 172.21.15.47:0/1851432164 >> 172.21.15.47:0/1851432164 conn(0x7f62f80f63c0 msgr2=0x7f62f80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:48:33.078 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.073+0000 7f62fd20d700 1 -- 172.21.15.47:0/1851432164 shutdown_connections 2024-01-18T06:48:33.079 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.073+0000 7f62fd20d700 1 -- 172.21.15.47:0/1851432164 wait complete. 2024-01-18T06:48:33.079 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.074+0000 7f62fd20d700 1 Processor -- start 2024-01-18T06:48:33.079 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.074+0000 7f62fd20d700 1 -- start start 2024-01-18T06:48:33.079 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.074+0000 7f62fd20d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62f8070390 0x7f62f810cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:33.079 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.074+0000 7f62fd20d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f810d1d0 con 0x7f62f8070390 2024-01-18T06:48:33.079 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.074+0000 7f62f6d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62f8070390 0x7f62f810cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:33.079 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.074+0000 7f62f6d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62f8070390 0x7f62f810cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36252/0 (socket says 172.21.15.47:36252) 2024-01-18T06:48:33.079 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.074+0000 7f62f6d9d700 1 -- 172.21.15.47:0/478874014 learned_addr learned my addr 172.21.15.47:0/478874014 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:48:33.079 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.074+0000 7f62f6d9d700 1 -- 172.21.15.47:0/478874014 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62ec00b200 con 0x7f62f8070390 2024-01-18T06:48:33.079 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.074+0000 7f62f6d9d700 1 --2- 172.21.15.47:0/478874014 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62f8070390 0x7f62f810cce0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f62ec00b1d0 tx=0x7f62ec006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:33.079 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.075+0000 7f62df7fe700 1 -- 172.21.15.47:0/478874014 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62ec011df0 con 0x7f62f8070390 2024-01-18T06:48:33.080 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.075+0000 7f62df7fe700 1 -- 172.21.15.47:0/478874014 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f62ec01b8c0 con 0x7f62f8070390 2024-01-18T06:48:33.080 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.075+0000 7f62fd20d700 1 -- 172.21.15.47:0/478874014 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62f810d3d0 con 0x7f62f8070390 2024-01-18T06:48:33.080 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.075+0000 7f62df7fe700 1 -- 172.21.15.47:0/478874014 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62ec01a830 con 0x7f62f8070390 2024-01-18T06:48:33.080 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.075+0000 7f62fd20d700 1 -- 172.21.15.47:0/478874014 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62f810d7e0 con 0x7f62f8070390 2024-01-18T06:48:33.080 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.076+0000 7f62df7fe700 1 -- 172.21.15.47:0/478874014 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f62ec00f070 con 0x7f62f8070390 2024-01-18T06:48:33.080 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.076+0000 7f62df7fe700 1 --2- 172.21.15.47:0/478874014 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f62e402feb0 0x7f62e4032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:33.080 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.076+0000 7f62df7fe700 1 -- 172.21.15.47:0/478874014 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f62ec0410d0 con 0x7f62f8070390 2024-01-18T06:48:33.080 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.076+0000 7f62fd20d700 1 -- 172.21.15.47:0/478874014 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62f8045ad0 con 0x7f62f8070390 2024-01-18T06:48:33.080 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.076+0000 7f62f659c700 1 --2- 172.21.15.47:0/478874014 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f62e402feb0 0x7f62e4032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:33.080 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.076+0000 7f62f659c700 1 --2- 172.21.15.47:0/478874014 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f62e402feb0 0x7f62e4032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f62e80096a0 tx=0x7f62e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:33.081 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.080+0000 7f62df7fe700 1 -- 172.21.15.47:0/478874014 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f62ec016070 con 0x7f62f8070390 2024-01-18T06:48:33.254 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.255+0000 7f62fd20d700 1 -- 172.21.15.47:0/478874014 --> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f62f8058f20 con 0x7f62e402feb0 2024-01-18T06:48:33.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:33 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:31.972925+0000 mgr.smithi047.fuhzzr (mgr.14162) 3 : audit [DBG] from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-01-18T06:48:33.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:33 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:32.022772+0000 mon.smithi047 (mon.0) 107 : audit [INF] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' 2024-01-18T06:48:33.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:33 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:32.024489+0000 mon.smithi047 (mon.0) 108 : audit [INF] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' 2024-01-18T06:48:33.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:33 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:48:32.032200+0000 mon.smithi047 (mon.0) 109 : cluster [DBG] mgrmap e16: smithi047.fuhzzr(active, since 2s) 2024-01-18T06:48:33.500 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.500+0000 7f62df7fe700 1 -- 172.21.15.47:0/478874014 <== mgr.14162 v2:172.21.15.47:6800/1933549322 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f62f8058f20 con 0x7f62e402feb0 2024-01-18T06:48:33.500 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$gWxybyknjMnn3xNR61Uz5u1pu1AESpVEqH.v9dbsHeTcNxjmtmAOG", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1705560513, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-01-18T06:48:33.502 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.502+0000 7f62fd20d700 1 -- 172.21.15.47:0/478874014 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f62e402feb0 msgr2=0x7f62e4032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:33.502 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.502+0000 7f62fd20d700 1 --2- 172.21.15.47:0/478874014 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f62e402feb0 0x7f62e4032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f62e80096a0 tx=0x7f62e8006b40).stop 2024-01-18T06:48:33.502 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.502+0000 7f62fd20d700 1 -- 172.21.15.47:0/478874014 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62f8070390 msgr2=0x7f62f810cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:33.502 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.502+0000 7f62fd20d700 1 --2- 172.21.15.47:0/478874014 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62f8070390 0x7f62f810cce0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f62ec00b1d0 tx=0x7f62ec006ee0).stop 2024-01-18T06:48:33.502 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.502+0000 7f62fd20d700 1 -- 172.21.15.47:0/478874014 shutdown_connections 2024-01-18T06:48:33.502 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.502+0000 7f62fd20d700 1 --2- 172.21.15.47:0/478874014 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f62e402feb0 0x7f62e4032370 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:33.503 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.502+0000 7f62fd20d700 1 --2- 172.21.15.47:0/478874014 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62f8070390 0x7f62f810cce0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:33.503 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.502+0000 7f62fd20d700 1 -- 172.21.15.47:0/478874014 >> 172.21.15.47:0/478874014 conn(0x7f62f80f63c0 msgr2=0x7f62f80f7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:48:33.503 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.502+0000 7f62fd20d700 1 -- 172.21.15.47:0/478874014 shutdown_connections 2024-01-18T06:48:33.503 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:33.502+0000 7f62fd20d700 1 -- 172.21.15.47:0/478874014 wait complete. 2024-01-18T06:48:33.853 INFO:teuthology.orchestra.run.smithi047.stderr:Fetching dashboard port number... 2024-01-18T06:48:33.853 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/653e4012-b5cd-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpck5s1jz7:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcg0aieo2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2024-01-18T06:48:34.171 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.167+0000 7f4250ad6700 1 Processor -- start 2024-01-18T06:48:34.171 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.167+0000 7f4250ad6700 1 -- start start 2024-01-18T06:48:34.171 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.168+0000 7f4250ad6700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f424c0703f0 0x7f424c0fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:34.171 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.168+0000 7f4250ad6700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f424c0fee10 con 0x7f424c0703f0 2024-01-18T06:48:34.171 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.168+0000 7f424a59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f424c0703f0 0x7f424c0fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:34.171 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.168+0000 7f424a59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f424c0703f0 0x7f424c0fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36258/0 (socket says 172.21.15.47:36258) 2024-01-18T06:48:34.171 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.168+0000 7f424a59c700 1 -- 172.21.15.47:0/1010138799 learned_addr learned my addr 172.21.15.47:0/1010138799 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:48:34.171 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.168+0000 7f424a59c700 1 -- 172.21.15.47:0/1010138799 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f424c0fef50 con 0x7f424c0703f0 2024-01-18T06:48:34.171 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.168+0000 7f424a59c700 1 --2- 172.21.15.47:0/1010138799 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f424c0703f0 0x7f424c0fe920 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f423c00b5f0 tx=0x7f423c006ee0).ready entity=mon.0 client_cookie=82166bf993be339a server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:34.171 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.168+0000 7f424959a700 1 -- 172.21.15.47:0/1010138799 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f423c00d600 con 0x7f424c0703f0 2024-01-18T06:48:34.171 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.168+0000 7f424959a700 1 -- 172.21.15.47:0/1010138799 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f423c00dc10 con 0x7f424c0703f0 2024-01-18T06:48:34.171 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.168+0000 7f424959a700 1 -- 172.21.15.47:0/1010138799 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f423c011cd0 con 0x7f424c0703f0 2024-01-18T06:48:34.172 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.169+0000 7f4250ad6700 1 -- 172.21.15.47:0/1010138799 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f424c0703f0 msgr2=0x7f424c0fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:34.172 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.169+0000 7f4250ad6700 1 --2- 172.21.15.47:0/1010138799 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f424c0703f0 0x7f424c0fe920 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f423c00b5f0 tx=0x7f423c006ee0).stop 2024-01-18T06:48:34.172 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.169+0000 7f4250ad6700 1 -- 172.21.15.47:0/1010138799 shutdown_connections 2024-01-18T06:48:34.172 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.169+0000 7f4250ad6700 1 --2- 172.21.15.47:0/1010138799 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f424c0703f0 0x7f424c0fe920 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:34.172 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.169+0000 7f4250ad6700 1 -- 172.21.15.47:0/1010138799 >> 172.21.15.47:0/1010138799 conn(0x7f424c0f63c0 msgr2=0x7f424c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:48:34.172 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.169+0000 7f4250ad6700 1 -- 172.21.15.47:0/1010138799 shutdown_connections 2024-01-18T06:48:34.172 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.169+0000 7f4250ad6700 1 -- 172.21.15.47:0/1010138799 wait complete. 2024-01-18T06:48:34.172 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.170+0000 7f4250ad6700 1 Processor -- start 2024-01-18T06:48:34.172 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.170+0000 7f4250ad6700 1 -- start start 2024-01-18T06:48:34.172 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.170+0000 7f4250ad6700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f424c0703f0 0x7f424c0fc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:34.172 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.170+0000 7f4250ad6700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f424c0fa820 con 0x7f424c0703f0 2024-01-18T06:48:34.172 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.170+0000 7f424a59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f424c0703f0 0x7f424c0fc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:34.172 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.170+0000 7f424a59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f424c0703f0 0x7f424c0fc3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36274/0 (socket says 172.21.15.47:36274) 2024-01-18T06:48:34.172 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.170+0000 7f424a59c700 1 -- 172.21.15.47:0/2938703305 learned_addr learned my addr 172.21.15.47:0/2938703305 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:48:34.173 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.171+0000 7f424a59c700 1 -- 172.21.15.47:0/2938703305 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f423c00b200 con 0x7f424c0703f0 2024-01-18T06:48:34.173 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.171+0000 7f424a59c700 1 --2- 172.21.15.47:0/2938703305 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f424c0703f0 0x7f424c0fc3a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f423c006a40 tx=0x7f423c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:34.173 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.171+0000 7f423affd700 1 -- 172.21.15.47:0/2938703305 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f423c011c90 con 0x7f424c0703f0 2024-01-18T06:48:34.173 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.171+0000 7f423affd700 1 -- 172.21.15.47:0/2938703305 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f423c01b8c0 con 0x7f424c0703f0 2024-01-18T06:48:34.173 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.171+0000 7f423affd700 1 -- 172.21.15.47:0/2938703305 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f423c01a8c0 con 0x7f424c0703f0 2024-01-18T06:48:34.173 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.171+0000 7f4250ad6700 1 -- 172.21.15.47:0/2938703305 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f424c0fa9c0 con 0x7f424c0703f0 2024-01-18T06:48:34.173 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.171+0000 7f4250ad6700 1 -- 172.21.15.47:0/2938703305 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f424c0fadd0 con 0x7f424c0703f0 2024-01-18T06:48:34.174 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.172+0000 7f4250ad6700 1 -- 172.21.15.47:0/2938703305 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f424c045ad0 con 0x7f424c0703f0 2024-01-18T06:48:34.174 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.172+0000 7f423affd700 1 -- 172.21.15.47:0/2938703305 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f423c00f070 con 0x7f424c0703f0 2024-01-18T06:48:34.174 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.172+0000 7f423affd700 1 --2- 172.21.15.47:0/2938703305 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f423002ff00 0x7f42300323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:34.174 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.172+0000 7f423affd700 1 -- 172.21.15.47:0/2938703305 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f423c040db0 con 0x7f424c0703f0 2024-01-18T06:48:34.174 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.172+0000 7f4249d9b700 1 --2- 172.21.15.47:0/2938703305 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f423002ff00 0x7f42300323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:34.174 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.173+0000 7f4249d9b700 1 --2- 172.21.15.47:0/2938703305 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f423002ff00 0x7f42300323c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f42400096a0 tx=0x7f4240006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:34.175 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.176+0000 7f423affd700 1 -- 172.21.15.47:0/2938703305 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f423c0196c0 con 0x7f424c0703f0 2024-01-18T06:48:34.333 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.333+0000 7f4250ad6700 1 -- 172.21.15.47:0/2938703305 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f424c104860 con 0x7f424c0703f0 2024-01-18T06:48:34.333 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.334+0000 7f423affd700 1 -- 172.21.15.47:0/2938703305 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v10) v1 ==== 112+0+5 (secure 0 0 0) 0x7f423c016070 con 0x7f424c0703f0 2024-01-18T06:48:34.333 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 8443 2024-01-18T06:48:34.336 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.335+0000 7f4250ad6700 1 -- 172.21.15.47:0/2938703305 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f423002ff00 msgr2=0x7f42300323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:34.336 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.335+0000 7f4250ad6700 1 --2- 172.21.15.47:0/2938703305 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f423002ff00 0x7f42300323c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f42400096a0 tx=0x7f4240006b40).stop 2024-01-18T06:48:34.336 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.335+0000 7f4250ad6700 1 -- 172.21.15.47:0/2938703305 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f424c0703f0 msgr2=0x7f424c0fc3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:34.336 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.335+0000 7f4250ad6700 1 --2- 172.21.15.47:0/2938703305 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f424c0703f0 0x7f424c0fc3a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f423c006a40 tx=0x7f423c006ee0).stop 2024-01-18T06:48:34.336 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.335+0000 7f4250ad6700 1 -- 172.21.15.47:0/2938703305 shutdown_connections 2024-01-18T06:48:34.336 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.335+0000 7f4250ad6700 1 --2- 172.21.15.47:0/2938703305 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f423002ff00 0x7f42300323c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:34.336 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.335+0000 7f4250ad6700 1 --2- 172.21.15.47:0/2938703305 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f424c0703f0 0x7f424c0fc3a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:34.336 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.335+0000 7f4250ad6700 1 -- 172.21.15.47:0/2938703305 >> 172.21.15.47:0/2938703305 conn(0x7f424c0f63c0 msgr2=0x7f424c0f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:48:34.337 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.336+0000 7f4250ad6700 1 -- 172.21.15.47:0/2938703305 shutdown_connections 2024-01-18T06:48:34.337 INFO:teuthology.orchestra.run.smithi047.stderr:/usr/bin/ceph: 2024-01-18T06:48:34.336+0000 7f4250ad6700 1 -- 172.21.15.47:0/2938703305 wait complete. 2024-01-18T06:48:34.663 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:34 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:33.256242+0000 mgr.smithi047.fuhzzr (mgr.14162) 4 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-01-18T06:48:34.663 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:34 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:33.500910+0000 mon.smithi047 (mon.0) 110 : audit [INF] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' 2024-01-18T06:48:34.663 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:34 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:34.334350+0000 mon.smithi047 (mon.0) 111 : audit [DBG] from='client.? 172.21.15.47:0/2938703305' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-01-18T06:48:34.686 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-18T06:48:34.694 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: disabled 2024-01-18T06:48:34.694 INFO:teuthology.orchestra.run.smithi047.stderr:Running command: systemctl is-active firewalld.service 2024-01-18T06:48:34.701 INFO:teuthology.orchestra.run.smithi047.stderr:systemctl: inactive 2024-01-18T06:48:34.702 INFO:teuthology.orchestra.run.smithi047.stderr:firewalld.service is not enabled 2024-01-18T06:48:34.702 INFO:teuthology.orchestra.run.smithi047.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-01-18T06:48:34.705 INFO:teuthology.orchestra.run.smithi047.stderr:Ceph Dashboard is now available at: 2024-01-18T06:48:34.706 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-01-18T06:48:34.706 INFO:teuthology.orchestra.run.smithi047.stderr: URL: https://smithi047.front.sepia.ceph.com:8443/ 2024-01-18T06:48:34.706 INFO:teuthology.orchestra.run.smithi047.stderr: User: admin 2024-01-18T06:48:34.706 INFO:teuthology.orchestra.run.smithi047.stderr: Password: aa0mpord9f 2024-01-18T06:48:34.706 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-01-18T06:48:34.706 INFO:teuthology.orchestra.run.smithi047.stderr:You can access the Ceph CLI with: 2024-01-18T06:48:34.706 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-01-18T06:48:34.706 INFO:teuthology.orchestra.run.smithi047.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-01-18T06:48:34.706 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-01-18T06:48:34.706 INFO:teuthology.orchestra.run.smithi047.stderr:Please consider enabling telemetry to help improve Ceph: 2024-01-18T06:48:34.706 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-01-18T06:48:34.706 INFO:teuthology.orchestra.run.smithi047.stderr: ceph telemetry on 2024-01-18T06:48:34.706 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-01-18T06:48:34.706 INFO:teuthology.orchestra.run.smithi047.stderr:For more information see: 2024-01-18T06:48:34.706 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-01-18T06:48:34.707 INFO:teuthology.orchestra.run.smithi047.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2024-01-18T06:48:34.707 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-01-18T06:48:34.707 INFO:teuthology.orchestra.run.smithi047.stderr:Bootstrap complete. 2024-01-18T06:48:34.707 INFO:teuthology.orchestra.run.smithi047.stderr:Releasing lock 140595555518952 on /run/cephadm/653e4012-b5cd-11ee-95ae-87774f69a715.lock 2024-01-18T06:48:34.707 INFO:teuthology.orchestra.run.smithi047.stderr:Lock 140595555518952 released on /run/cephadm/653e4012-b5cd-11ee-95ae-87774f69a715.lock 2024-01-18T06:48:34.740 INFO:tasks.cephadm:Fetching config... 2024-01-18T06:48:34.740 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-01-18T06:48:34.740 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-01-18T06:48:34.799 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-01-18T06:48:34.800 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-01-18T06:48:34.800 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-01-18T06:48:34.860 INFO:tasks.cephadm:Fetching mon keyring... 2024-01-18T06:48:34.861 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-01-18T06:48:34.861 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/mon.smithi047/keyring of=/dev/stdout 2024-01-18T06:48:34.929 INFO:tasks.cephadm:Fetching pub ssh key... 2024-01-18T06:48:34.929 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-01-18T06:48:34.929 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-01-18T06:48:34.987 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-01-18T06:48:34.988 DEBUG:teuthology.orchestra.run.smithi047:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTUlFvL9QJzLbw5AICxcbAnlSDLfp3OyWF2qfKGQW5xqA24NQD/ytwqGXzFdU7CQB6S/43aYjhvz1aoYDblZCYmkQ0zmszLQRF3qzQ88zfyWrZrG6Yjl5L0sA1S8zQkd9fSfXXN0C5Zx5NYa3bfS59JmCoRWYX6Hm0t/IokIGAwQBu+ti0GTCrJvnkrMSRoAjny3OtFQ3DSsrOgzsLBOM8Ij3xky25fhFunkjzMLBlPTT3IogH2j5fwlr9XCk4ro8k+N735KuMN1JS8PJiNVpJWV+CYVybH1gMGK0hPelKMzARPJu4fi6nFeR7jQ3DxXzqq5O+qPWmH8JfMDSHXM+eMbKommvet1WW2b5GsmkTx5TplWn1LrplbMb1NPSzJC/7Zz04Tg9Bg3Fc38xk6vLXTMQYYXzCTOD8MY26IX07qq5KKDNCImtxdR0+Bnor7WSTJK5yYGYUG9Jy29mAXrRUoKdZAc1KxngjQkejKQCyx4YxplK6RoKGKmV3KnZ+des= ceph-653e4012-b5cd-11ee-95ae-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-18T06:48:35.065 INFO:teuthology.orchestra.run.smithi047.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTUlFvL9QJzLbw5AICxcbAnlSDLfp3OyWF2qfKGQW5xqA24NQD/ytwqGXzFdU7CQB6S/43aYjhvz1aoYDblZCYmkQ0zmszLQRF3qzQ88zfyWrZrG6Yjl5L0sA1S8zQkd9fSfXXN0C5Zx5NYa3bfS59JmCoRWYX6Hm0t/IokIGAwQBu+ti0GTCrJvnkrMSRoAjny3OtFQ3DSsrOgzsLBOM8Ij3xky25fhFunkjzMLBlPTT3IogH2j5fwlr9XCk4ro8k+N735KuMN1JS8PJiNVpJWV+CYVybH1gMGK0hPelKMzARPJu4fi6nFeR7jQ3DxXzqq5O+qPWmH8JfMDSHXM+eMbKommvet1WW2b5GsmkTx5TplWn1LrplbMb1NPSzJC/7Zz04Tg9Bg3Fc38xk6vLXTMQYYXzCTOD8MY26IX07qq5KKDNCImtxdR0+Bnor7WSTJK5yYGYUG9Jy29mAXrRUoKdZAc1KxngjQkejKQCyx4YxplK6RoKGKmV3KnZ+des= ceph-653e4012-b5cd-11ee-95ae-87774f69a715 2024-01-18T06:48:35.077 DEBUG:teuthology.orchestra.run.smithi194:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTUlFvL9QJzLbw5AICxcbAnlSDLfp3OyWF2qfKGQW5xqA24NQD/ytwqGXzFdU7CQB6S/43aYjhvz1aoYDblZCYmkQ0zmszLQRF3qzQ88zfyWrZrG6Yjl5L0sA1S8zQkd9fSfXXN0C5Zx5NYa3bfS59JmCoRWYX6Hm0t/IokIGAwQBu+ti0GTCrJvnkrMSRoAjny3OtFQ3DSsrOgzsLBOM8Ij3xky25fhFunkjzMLBlPTT3IogH2j5fwlr9XCk4ro8k+N735KuMN1JS8PJiNVpJWV+CYVybH1gMGK0hPelKMzARPJu4fi6nFeR7jQ3DxXzqq5O+qPWmH8JfMDSHXM+eMbKommvet1WW2b5GsmkTx5TplWn1LrplbMb1NPSzJC/7Zz04Tg9Bg3Fc38xk6vLXTMQYYXzCTOD8MY26IX07qq5KKDNCImtxdR0+Bnor7WSTJK5yYGYUG9Jy29mAXrRUoKdZAc1KxngjQkejKQCyx4YxplK6RoKGKmV3KnZ+des= ceph-653e4012-b5cd-11ee-95ae-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-18T06:48:35.123 INFO:teuthology.orchestra.run.smithi194.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTUlFvL9QJzLbw5AICxcbAnlSDLfp3OyWF2qfKGQW5xqA24NQD/ytwqGXzFdU7CQB6S/43aYjhvz1aoYDblZCYmkQ0zmszLQRF3qzQ88zfyWrZrG6Yjl5L0sA1S8zQkd9fSfXXN0C5Zx5NYa3bfS59JmCoRWYX6Hm0t/IokIGAwQBu+ti0GTCrJvnkrMSRoAjny3OtFQ3DSsrOgzsLBOM8Ij3xky25fhFunkjzMLBlPTT3IogH2j5fwlr9XCk4ro8k+N735KuMN1JS8PJiNVpJWV+CYVybH1gMGK0hPelKMzARPJu4fi6nFeR7jQ3DxXzqq5O+qPWmH8JfMDSHXM+eMbKommvet1WW2b5GsmkTx5TplWn1LrplbMb1NPSzJC/7Zz04Tg9Bg3Fc38xk6vLXTMQYYXzCTOD8MY26IX07qq5KKDNCImtxdR0+Bnor7WSTJK5yYGYUG9Jy29mAXrRUoKdZAc1KxngjQkejKQCyx4YxplK6RoKGKmV3KnZ+des= ceph-653e4012-b5cd-11ee-95ae-87774f69a715 2024-01-18T06:48:35.135 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-01-18T06:48:36.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:48:36.720+0000 7f60b490b700 1 -- 172.21.15.47:0/1714923464 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60b0060280 msgr2=0x7f60b00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:36.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:48:36.720+0000 7f60b490b700 1 --2- 172.21.15.47:0/1714923464 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60b0060280 0x7f60b00f90f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f60a0009fb0 tx=0x7f60a0009b30).stop 2024-01-18T06:48:36.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:48:36.720+0000 7f60b490b700 1 -- 172.21.15.47:0/1714923464 shutdown_connections 2024-01-18T06:48:36.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:48:36.720+0000 7f60b490b700 1 --2- 172.21.15.47:0/1714923464 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60b0060280 0x7f60b00f90f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:36.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:48:36.720+0000 7f60b490b700 1 -- 172.21.15.47:0/1714923464 >> 172.21.15.47:0/1714923464 conn(0x7f60b00f0b70 msgr2=0x7f60b00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:48:36.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:48:36.720+0000 7f60b490b700 1 -- 172.21.15.47:0/1714923464 shutdown_connections 2024-01-18T06:48:36.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:48:36.720+0000 7f60b490b700 1 -- 172.21.15.47:0/1714923464 wait complete. 2024-01-18T06:48:36.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:48:36.721+0000 7f60b490b700 1 Processor -- start 2024-01-18T06:48:36.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:48:36.721+0000 7f60b490b700 1 -- start start 2024-01-18T06:48:36.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:48:36.721+0000 7f60b490b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60b0060280 0x7f60b01066b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:36.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:48:36.721+0000 7f60b490b700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b0106ba0 con 0x7f60b0060280 2024-01-18T06:48:36.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:48:36.721+0000 7f60ae59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60b0060280 0x7f60b01066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:36.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:48:36.721+0000 7f60ae59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60b0060280 0x7f60b01066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:36284/0 (socket says 172.21.15.47:36284) 2024-01-18T06:48:36.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:48:36.721+0000 7f60ae59c700 1 -- 172.21.15.47:0/2623071044 learned_addr learned my addr 172.21.15.47:0/2623071044 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:48:36.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:48:36.722+0000 7f60ae59c700 1 -- 172.21.15.47:0/2623071044 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60a0005040 con 0x7f60b0060280 2024-01-18T06:48:36.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:48:36.722+0000 7f60ae59c700 1 --2- 172.21.15.47:0/2623071044 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60b0060280 0x7f60b01066b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f60a0013510 tx=0x7f60a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:36.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:48:36.722+0000 7f609effd700 1 -- 172.21.15.47:0/2623071044 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60a0009320 con 0x7f60b0060280 2024-01-18T06:48:36.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:48:36.722+0000 7f60b490b700 1 -- 172.21.15.47:0/2623071044 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60b0106da0 con 0x7f60b0060280 2024-01-18T06:48:36.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:48:36.722+0000 7f609effd700 1 -- 172.21.15.47:0/2623071044 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f60a0013f00 con 0x7f60b0060280 2024-01-18T06:48:36.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:48:36.722+0000 7f609effd700 1 -- 172.21.15.47:0/2623071044 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60a0014dc0 con 0x7f60b0060280 2024-01-18T06:48:36.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:48:36.722+0000 7f60b490b700 1 -- 172.21.15.47:0/2623071044 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60b0107230 con 0x7f60b0060280 2024-01-18T06:48:36.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:48:36.723+0000 7f609effd700 1 -- 172.21.15.47:0/2623071044 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f60a001d3d0 con 0x7f60b0060280 2024-01-18T06:48:36.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:48:36.723+0000 7f609effd700 1 --2- 172.21.15.47:0/2623071044 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f609802ff60 0x7f6098032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:36.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:48:36.723+0000 7f609effd700 1 -- 172.21.15.47:0/2623071044 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f60a0041040 con 0x7f60b0060280 2024-01-18T06:48:36.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:48:36.723+0000 7f60b490b700 1 -- 172.21.15.47:0/2623071044 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60b0045ad0 con 0x7f60b0060280 2024-01-18T06:48:36.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:48:36.723+0000 7f60add9b700 1 --2- 172.21.15.47:0/2623071044 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f609802ff60 0x7f6098032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:36.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:48:36.723+0000 7f60add9b700 1 --2- 172.21.15.47:0/2623071044 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f609802ff60 0x7f6098032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f60a40096a0 tx=0x7f60a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:36.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:48:36.727+0000 7f609effd700 1 -- 172.21.15.47:0/2623071044 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f60a0012920 con 0x7f60b0060280 2024-01-18T06:48:36.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:48:36.888+0000 7f60b490b700 1 -- 172.21.15.47:0/2623071044 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f60b0108310 con 0x7f60b0060280 2024-01-18T06:48:36.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:48:36.892+0000 7f609effd700 1 -- 172.21.15.47:0/2623071044 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v11)=0 v11) v1 ==== 125+0+0 (secure 0 0 0) 0x7f60a002b020 con 0x7f60b0060280 2024-01-18T06:48:36.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:48:36.894+0000 7f60b490b700 1 -- 172.21.15.47:0/2623071044 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f609802ff60 msgr2=0x7f6098032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:36.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:48:36.894+0000 7f60b490b700 1 --2- 172.21.15.47:0/2623071044 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f609802ff60 0x7f6098032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f60a40096a0 tx=0x7f60a4006b40).stop 2024-01-18T06:48:36.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:48:36.894+0000 7f60b490b700 1 -- 172.21.15.47:0/2623071044 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60b0060280 msgr2=0x7f60b01066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:36.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:48:36.894+0000 7f60b490b700 1 --2- 172.21.15.47:0/2623071044 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60b0060280 0x7f60b01066b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f60a0013510 tx=0x7f60a000b040).stop 2024-01-18T06:48:36.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:48:36.894+0000 7f60b490b700 1 -- 172.21.15.47:0/2623071044 shutdown_connections 2024-01-18T06:48:36.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:48:36.894+0000 7f60b490b700 1 --2- 172.21.15.47:0/2623071044 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f609802ff60 0x7f6098032420 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:36.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:48:36.894+0000 7f60b490b700 1 --2- 172.21.15.47:0/2623071044 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60b0060280 0x7f60b01066b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:36.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:48:36.894+0000 7f60b490b700 1 -- 172.21.15.47:0/2623071044 >> 172.21.15.47:0/2623071044 conn(0x7f60b00f0b70 msgr2=0x7f60b00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:48:36.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:48:36.894+0000 7f60b490b700 1 -- 172.21.15.47:0/2623071044 shutdown_connections 2024-01-18T06:48:36.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:48:36.894+0000 7f60b490b700 1 -- 172.21.15.47:0/2623071044 wait complete. 2024-01-18T06:48:37.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:36 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:48:35.994758+0000 mon.smithi047 (mon.0) 112 : cluster [DBG] mgrmap e17: smithi047.fuhzzr(active, since 6s) 2024-01-18T06:48:37.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:36 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:36.787969+0000 mon.smithi047 (mon.0) 113 : audit [INF] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' 2024-01-18T06:48:37.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:36 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:36.892386+0000 mon.smithi047 (mon.0) 114 : audit [INF] from='client.? 172.21.15.47:0/2623071044' entity='client.admin' 2024-01-18T06:48:37.283 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi194 2024-01-18T06:48:37.284 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-01-18T06:48:37.284 DEBUG:teuthology.orchestra.run.smithi194:> dd of=/etc/ceph/ceph.conf 2024-01-18T06:48:37.306 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-01-18T06:48:37.306 DEBUG:teuthology.orchestra.run.smithi194:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-18T06:48:37.367 INFO:tasks.cephadm:Adding host smithi194 to orchestrator... 2024-01-18T06:48:37.367 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph orch host add smithi194 2024-01-18T06:48:38.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:38 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:37.188604+0000 mon.smithi047 (mon.0) 115 : audit [INF] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' 2024-01-18T06:48:38.464 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:38 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cephadm 2024-01-18T06:48:37.195153+0000 mgr.smithi047.fuhzzr (mgr.14162) 5 : cephadm [INF] Deploying daemon alertmanager.smithi047 on smithi047 2024-01-18T06:48:51.464 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:50 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:48:49.974648+0000 mgr.smithi047.fuhzzr (mgr.14162) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:48:53.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:52 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:48:51.975002+0000 mgr.smithi047.fuhzzr (mgr.14162) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:48:55.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:54 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:48:53.975292+0000 mgr.smithi047.fuhzzr (mgr.14162) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:48:57.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:48:55.975537+0000 mgr.smithi047.fuhzzr (mgr.14162) 9 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:48:58.931 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:48:58.929+0000 7f47e3566700 1 -- 172.21.15.194:0/970886822 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f47dc0f6510 msgr2=0x7f47dc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:58.931 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:48:58.929+0000 7f47e3566700 1 --2- 172.21.15.194:0/970886822 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f47dc0f6510 0x7f47dc0f6930 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f47cc009fb0 tx=0x7f47cc009b30).stop 2024-01-18T06:48:58.931 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:48:58.929+0000 7f47e3566700 1 -- 172.21.15.194:0/970886822 shutdown_connections 2024-01-18T06:48:58.931 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:48:58.929+0000 7f47e3566700 1 --2- 172.21.15.194:0/970886822 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f47dc0f6510 0x7f47dc0f6930 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:58.931 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:48:58.929+0000 7f47e3566700 1 -- 172.21.15.194:0/970886822 >> 172.21.15.194:0/970886822 conn(0x7f47dc0f1c70 msgr2=0x7f47dc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:48:58.931 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:48:58.929+0000 7f47e3566700 1 -- 172.21.15.194:0/970886822 shutdown_connections 2024-01-18T06:48:58.931 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:48:58.930+0000 7f47e3566700 1 -- 172.21.15.194:0/970886822 wait complete. 2024-01-18T06:48:58.931 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:48:58.930+0000 7f47e3566700 1 Processor -- start 2024-01-18T06:48:58.932 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:48:58.930+0000 7f47e3566700 1 -- start start 2024-01-18T06:48:58.932 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:48:58.930+0000 7f47e3566700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f47dc0f6510 0x7f47dc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:58.932 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:48:58.930+0000 7f47e3566700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47dc106c20 con 0x7f47dc0f6510 2024-01-18T06:48:58.932 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:48:58.930+0000 7f47e1302700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f47dc0f6510 0x7f47dc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:58.932 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:48:58.930+0000 7f47e1302700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f47dc0f6510 0x7f47dc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:52844/0 (socket says 172.21.15.194:52844) 2024-01-18T06:48:58.932 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:48:58.930+0000 7f47e1302700 1 -- 172.21.15.194:0/2998249355 learned_addr learned my addr 172.21.15.194:0/2998249355 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:48:58.932 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:48:58.931+0000 7f47e1302700 1 -- 172.21.15.194:0/2998249355 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47cc005040 con 0x7f47dc0f6510 2024-01-18T06:48:58.932 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:48:58.931+0000 7f47e1302700 1 --2- 172.21.15.194:0/2998249355 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f47dc0f6510 0x7f47dc106730 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f47cc013510 tx=0x7f47cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:58.932 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:48:58.931+0000 7f47d1ffb700 1 -- 172.21.15.194:0/2998249355 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47cc009320 con 0x7f47dc0f6510 2024-01-18T06:48:58.932 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:48:58.931+0000 7f47e3566700 1 -- 172.21.15.194:0/2998249355 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47dc106e20 con 0x7f47dc0f6510 2024-01-18T06:48:58.933 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:48:58.931+0000 7f47d1ffb700 1 -- 172.21.15.194:0/2998249355 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f47cc013f00 con 0x7f47dc0f6510 2024-01-18T06:48:58.933 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:48:58.931+0000 7f47d1ffb700 1 -- 172.21.15.194:0/2998249355 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47cc014d90 con 0x7f47dc0f6510 2024-01-18T06:48:58.933 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:48:58.931+0000 7f47e3566700 1 -- 172.21.15.194:0/2998249355 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47dc1072b0 con 0x7f47dc0f6510 2024-01-18T06:48:58.934 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:48:58.932+0000 7f47d1ffb700 1 -- 172.21.15.194:0/2998249355 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f47cc0263a0 con 0x7f47dc0f6510 2024-01-18T06:48:58.934 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:48:58.932+0000 7f47d1ffb700 1 --2- 172.21.15.194:0/2998249355 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f47c802ff60 0x7f47c8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:48:58.934 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:48:58.932+0000 7f47e3566700 1 -- 172.21.15.194:0/2998249355 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47dc045ad0 con 0x7f47dc0f6510 2024-01-18T06:48:58.934 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:48:58.932+0000 7f47d1ffb700 1 -- 172.21.15.194:0/2998249355 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f47cc0410b0 con 0x7f47dc0f6510 2024-01-18T06:48:58.934 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:48:58.932+0000 7f47e0b01700 1 --2- 172.21.15.194:0/2998249355 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f47c802ff60 0x7f47c8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:48:58.934 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:48:58.933+0000 7f47e0b01700 1 --2- 172.21.15.194:0/2998249355 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f47c802ff60 0x7f47c8032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f47d80096a0 tx=0x7f47d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:48:58.936 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:48:58.937+0000 7f47d1ffb700 1 -- 172.21.15.194:0/2998249355 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f47cc012920 con 0x7f47dc0f6510 2024-01-18T06:48:59.110 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:48:59.110+0000 7f47e3566700 1 -- 172.21.15.194:0/2998249355 --> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi194", "target": ["mon-mgr", ""]}) v1 -- 0x7f47dc0f2e70 con 0x7f47c802ff60 2024-01-18T06:48:59.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:48:58 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:48:57.975831+0000 mgr.smithi047.fuhzzr (mgr.14162) 10 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:48:59.705 INFO:teuthology.orchestra.run.smithi194.stdout:Added host 'smithi194' 2024-01-18T06:48:59.705 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:48:59.703+0000 7f47d1ffb700 1 -- 172.21.15.194:0/2998249355 <== mgr.14162 v2:172.21.15.47:6800/1933549322 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f47dc0f2e70 con 0x7f47c802ff60 2024-01-18T06:48:59.706 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:48:59.705+0000 7f47e3566700 1 -- 172.21.15.194:0/2998249355 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f47c802ff60 msgr2=0x7f47c8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:59.706 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:48:59.705+0000 7f47e3566700 1 --2- 172.21.15.194:0/2998249355 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f47c802ff60 0x7f47c8032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f47d80096a0 tx=0x7f47d8006b40).stop 2024-01-18T06:48:59.706 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:48:59.705+0000 7f47e3566700 1 -- 172.21.15.194:0/2998249355 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f47dc0f6510 msgr2=0x7f47dc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:48:59.707 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:48:59.705+0000 7f47e3566700 1 --2- 172.21.15.194:0/2998249355 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f47dc0f6510 0x7f47dc106730 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f47cc013510 tx=0x7f47cc00b040).stop 2024-01-18T06:48:59.707 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:48:59.706+0000 7f47e3566700 1 -- 172.21.15.194:0/2998249355 shutdown_connections 2024-01-18T06:48:59.707 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:48:59.706+0000 7f47e3566700 1 --2- 172.21.15.194:0/2998249355 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f47c802ff60 0x7f47c8032420 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:59.707 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:48:59.706+0000 7f47e3566700 1 --2- 172.21.15.194:0/2998249355 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f47dc0f6510 0x7f47dc106730 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:48:59.707 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:48:59.706+0000 7f47e3566700 1 -- 172.21.15.194:0/2998249355 >> 172.21.15.194:0/2998249355 conn(0x7f47dc0f1c70 msgr2=0x7f47dc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:48:59.707 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:48:59.706+0000 7f47e3566700 1 -- 172.21.15.194:0/2998249355 shutdown_connections 2024-01-18T06:48:59.707 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:48:59.706+0000 7f47e3566700 1 -- 172.21.15.194:0/2998249355 wait complete. 2024-01-18T06:49:00.072 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph orch host ls --format=json 2024-01-18T06:49:00.963 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:00 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:59.111627+0000 mgr.smithi047.fuhzzr (mgr.14162) 11 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi194", "target": ["mon-mgr", ""]}]: dispatch 2024-01-18T06:49:00.963 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:00 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:48:59.704212+0000 mon.smithi047 (mon.0) 116 : audit [INF] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' 2024-01-18T06:49:00.963 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:00 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cephadm 2024-01-18T06:48:59.704679+0000 mgr.smithi047.fuhzzr (mgr.14162) 12 : cephadm [INF] Added host smithi194 2024-01-18T06:49:01.234 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:01.225+0000 7fbd49ecc700 1 -- 172.21.15.194:0/3008458998 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd440f5410 msgr2=0x7fbd440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:01.234 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:01.225+0000 7fbd49ecc700 1 --2- 172.21.15.194:0/3008458998 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd440f5410 0x7fbd440f5830 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fbd34009fb0 tx=0x7fbd34009b30).stop 2024-01-18T06:49:01.234 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:01.225+0000 7fbd49ecc700 1 -- 172.21.15.194:0/3008458998 shutdown_connections 2024-01-18T06:49:01.234 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:01.225+0000 7fbd49ecc700 1 --2- 172.21.15.194:0/3008458998 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd440f5410 0x7fbd440f5830 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:01.234 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:01.225+0000 7fbd49ecc700 1 -- 172.21.15.194:0/3008458998 >> 172.21.15.194:0/3008458998 conn(0x7fbd440f0b70 msgr2=0x7fbd440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:49:01.234 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:01.225+0000 7fbd49ecc700 1 -- 172.21.15.194:0/3008458998 shutdown_connections 2024-01-18T06:49:01.234 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:01.225+0000 7fbd49ecc700 1 -- 172.21.15.194:0/3008458998 wait complete. 2024-01-18T06:49:01.234 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:01.226+0000 7fbd49ecc700 1 Processor -- start 2024-01-18T06:49:01.234 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:01.226+0000 7fbd49ecc700 1 -- start start 2024-01-18T06:49:01.235 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:01.226+0000 7fbd49ecc700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd440f5410 0x7fbd44109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:49:01.235 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:01.226+0000 7fbd49ecc700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd44060240 con 0x7fbd440f5410 2024-01-18T06:49:01.235 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:01.226+0000 7fbd437fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd440f5410 0x7fbd44109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:49:01.235 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:01.226+0000 7fbd437fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd440f5410 0x7fbd44109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:52858/0 (socket says 172.21.15.194:52858) 2024-01-18T06:49:01.235 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:01.226+0000 7fbd437fe700 1 -- 172.21.15.194:0/4162432353 learned_addr learned my addr 172.21.15.194:0/4162432353 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:49:01.235 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:01.227+0000 7fbd437fe700 1 -- 172.21.15.194:0/4162432353 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd34005040 con 0x7fbd440f5410 2024-01-18T06:49:01.235 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:01.227+0000 7fbd437fe700 1 --2- 172.21.15.194:0/4162432353 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd440f5410 0x7fbd44109430 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fbd34013510 tx=0x7fbd3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:49:01.235 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:01.227+0000 7fbd3bfff700 1 -- 172.21.15.194:0/4162432353 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd34009320 con 0x7fbd440f5410 2024-01-18T06:49:01.235 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:01.227+0000 7fbd49ecc700 1 -- 172.21.15.194:0/4162432353 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd44060440 con 0x7fbd440f5410 2024-01-18T06:49:01.236 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:01.227+0000 7fbd3bfff700 1 -- 172.21.15.194:0/4162432353 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbd34013f00 con 0x7fbd440f5410 2024-01-18T06:49:01.236 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:01.227+0000 7fbd3bfff700 1 -- 172.21.15.194:0/4162432353 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd34014d90 con 0x7fbd440f5410 2024-01-18T06:49:01.236 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:01.227+0000 7fbd49ecc700 1 -- 172.21.15.194:0/4162432353 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd44109ae0 con 0x7fbd440f5410 2024-01-18T06:49:01.236 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:01.228+0000 7fbd3bfff700 1 -- 172.21.15.194:0/4162432353 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbd340095f0 con 0x7fbd440f5410 2024-01-18T06:49:01.236 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:01.228+0000 7fbd3bfff700 1 --2- 172.21.15.194:0/4162432353 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fbd3002ff60 0x7fbd30032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:49:01.236 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:01.228+0000 7fbd49ecc700 1 -- 172.21.15.194:0/4162432353 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd44045ad0 con 0x7fbd440f5410 2024-01-18T06:49:01.236 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:01.228+0000 7fbd3bfff700 1 -- 172.21.15.194:0/4162432353 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbd34041000 con 0x7fbd440f5410 2024-01-18T06:49:01.236 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:01.228+0000 7fbd42ffd700 1 --2- 172.21.15.194:0/4162432353 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fbd3002ff60 0x7fbd30032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:49:01.237 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:01.229+0000 7fbd42ffd700 1 --2- 172.21.15.194:0/4162432353 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fbd3002ff60 0x7fbd30032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fbd2c0096a0 tx=0x7fbd2c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:49:01.237 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:01.232+0000 7fbd3bfff700 1 -- 172.21.15.194:0/4162432353 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbd34014420 con 0x7fbd440f5410 2024-01-18T06:49:01.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:01.404+0000 7fbd49ecc700 1 -- 172.21.15.194:0/4162432353 --> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbd440f1d70 con 0x7fbd3002ff60 2024-01-18T06:49:01.406 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:01.405+0000 7fbd3bfff700 1 -- 172.21.15.194:0/4162432353 <== mgr.14162 v2:172.21.15.47:6800/1933549322 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+156 (secure 0 0 0) 0x7fbd440f1d70 con 0x7fbd3002ff60 2024-01-18T06:49:01.407 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:49:01.407 INFO:teuthology.orchestra.run.smithi194.stdout:[{"addr": "172.21.15.47", "hostname": "smithi047", "labels": [], "status": ""}, {"addr": "smithi194", "hostname": "smithi194", "labels": [], "status": ""}] 2024-01-18T06:49:01.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:01.407+0000 7fbd49ecc700 1 -- 172.21.15.194:0/4162432353 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fbd3002ff60 msgr2=0x7fbd30032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:01.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:01.407+0000 7fbd49ecc700 1 --2- 172.21.15.194:0/4162432353 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fbd3002ff60 0x7fbd30032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fbd2c0096a0 tx=0x7fbd2c006b40).stop 2024-01-18T06:49:01.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:01.407+0000 7fbd49ecc700 1 -- 172.21.15.194:0/4162432353 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd440f5410 msgr2=0x7fbd44109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:01.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:01.407+0000 7fbd49ecc700 1 --2- 172.21.15.194:0/4162432353 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd440f5410 0x7fbd44109430 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fbd34013510 tx=0x7fbd3400b040).stop 2024-01-18T06:49:01.408 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:01.407+0000 7fbd49ecc700 1 -- 172.21.15.194:0/4162432353 shutdown_connections 2024-01-18T06:49:01.408 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:01.407+0000 7fbd49ecc700 1 --2- 172.21.15.194:0/4162432353 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fbd3002ff60 0x7fbd30032420 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:01.408 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:01.407+0000 7fbd49ecc700 1 --2- 172.21.15.194:0/4162432353 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd440f5410 0x7fbd44109430 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:01.408 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:01.407+0000 7fbd49ecc700 1 -- 172.21.15.194:0/4162432353 >> 172.21.15.194:0/4162432353 conn(0x7fbd440f0b70 msgr2=0x7fbd440f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:49:01.409 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:01.407+0000 7fbd49ecc700 1 -- 172.21.15.194:0/4162432353 shutdown_connections 2024-01-18T06:49:01.409 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:01.407+0000 7fbd49ecc700 1 -- 172.21.15.194:0/4162432353 wait complete. 2024-01-18T06:49:01.786 INFO:tasks.cephadm:Setting crush tunables to default 2024-01-18T06:49:01.787 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph osd crush tunables default 2024-01-18T06:49:01.963 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:01 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:48:59.976119+0000 mgr.smithi047.fuhzzr (mgr.14162) 13 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:49:02.841 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:02 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:49:01.406918+0000 mgr.smithi047.fuhzzr (mgr.14162) 14 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-18T06:49:03.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:49:03.015+0000 7fcdb1d57700 1 -- 172.21.15.47:0/1630579234 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcdac0f69d0 msgr2=0x7fcdac0f6df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:03.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:49:03.015+0000 7fcdb1d57700 1 --2- 172.21.15.47:0/1630579234 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcdac0f69d0 0x7fcdac0f6df0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fcd9c009fb0 tx=0x7fcd9c009b30).stop 2024-01-18T06:49:03.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:49:03.015+0000 7fcdb1d57700 1 -- 172.21.15.47:0/1630579234 shutdown_connections 2024-01-18T06:49:03.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:49:03.015+0000 7fcdb1d57700 1 --2- 172.21.15.47:0/1630579234 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcdac0f69d0 0x7fcdac0f6df0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:03.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:49:03.015+0000 7fcdb1d57700 1 -- 172.21.15.47:0/1630579234 >> 172.21.15.47:0/1630579234 conn(0x7fcdac0f02e0 msgr2=0x7fcdac0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:49:03.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:49:03.015+0000 7fcdb1d57700 1 -- 172.21.15.47:0/1630579234 shutdown_connections 2024-01-18T06:49:03.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:49:03.015+0000 7fcdb1d57700 1 -- 172.21.15.47:0/1630579234 wait complete. 2024-01-18T06:49:03.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:49:03.015+0000 7fcdb1d57700 1 Processor -- start 2024-01-18T06:49:03.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:49:03.015+0000 7fcdb1d57700 1 -- start start 2024-01-18T06:49:03.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:49:03.016+0000 7fcdb1d57700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcdac0f69d0 0x7fcdac0fd390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:49:03.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:49:03.016+0000 7fcdb1d57700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac0fb810 con 0x7fcdac0f69d0 2024-01-18T06:49:03.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:49:03.016+0000 7fcdab7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcdac0f69d0 0x7fcdac0fd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:49:03.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:49:03.016+0000 7fcdab7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcdac0f69d0 0x7fcdac0fd390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:58884/0 (socket says 172.21.15.47:58884) 2024-01-18T06:49:03.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:49:03.016+0000 7fcdab7fe700 1 -- 172.21.15.47:0/438697654 learned_addr learned my addr 172.21.15.47:0/438697654 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-01-18T06:49:03.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:49:03.016+0000 7fcdab7fe700 1 -- 172.21.15.47:0/438697654 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd9c005040 con 0x7fcdac0f69d0 2024-01-18T06:49:03.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:49:03.016+0000 7fcdab7fe700 1 --2- 172.21.15.47:0/438697654 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcdac0f69d0 0x7fcdac0fd390 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fcd9c013510 tx=0x7fcd9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:49:03.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:49:03.016+0000 7fcda3fff700 1 -- 172.21.15.47:0/438697654 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd9c009320 con 0x7fcdac0f69d0 2024-01-18T06:49:03.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:49:03.016+0000 7fcda3fff700 1 -- 172.21.15.47:0/438697654 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcd9c013f00 con 0x7fcdac0f69d0 2024-01-18T06:49:03.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:49:03.016+0000 7fcdb1d57700 1 -- 172.21.15.47:0/438697654 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdac0fb9b0 con 0x7fcdac0f69d0 2024-01-18T06:49:03.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:49:03.016+0000 7fcda3fff700 1 -- 172.21.15.47:0/438697654 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd9c014d90 con 0x7fcdac0f69d0 2024-01-18T06:49:03.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:49:03.016+0000 7fcdb1d57700 1 -- 172.21.15.47:0/438697654 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdac0fbe40 con 0x7fcdac0f69d0 2024-01-18T06:49:03.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:49:03.017+0000 7fcda3fff700 1 -- 172.21.15.47:0/438697654 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcd9c0263a0 con 0x7fcdac0f69d0 2024-01-18T06:49:03.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:49:03.017+0000 7fcda3fff700 1 --2- 172.21.15.47:0/438697654 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fcd9802ff60 0x7fcd98032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:49:03.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:49:03.017+0000 7fcda3fff700 1 -- 172.21.15.47:0/438697654 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcd9c041fb0 con 0x7fcdac0f69d0 2024-01-18T06:49:03.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:49:03.017+0000 7fcdaaffd700 1 --2- 172.21.15.47:0/438697654 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fcd9802ff60 0x7fcd98032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:49:03.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:49:03.017+0000 7fcdb1d57700 1 -- 172.21.15.47:0/438697654 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdac059100 con 0x7fcdac0f69d0 2024-01-18T06:49:03.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:49:03.018+0000 7fcdaaffd700 1 --2- 172.21.15.47:0/438697654 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fcd9802ff60 0x7fcd98032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fcd940096a0 tx=0x7fcd94006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:49:03.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:49:03.022+0000 7fcda3fff700 1 -- 172.21.15.47:0/438697654 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcd9c012920 con 0x7fcdac0f69d0 2024-01-18T06:49:03.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:49:03.187+0000 7fcdb1d57700 1 -- 172.21.15.47:0/438697654 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fcdac045ad0 con 0x7fcdac0f69d0 2024-01-18T06:49:03.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:49:03.915+0000 7fcda3fff700 1 -- 172.21.15.47:0/438697654 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7fcd9c02b030 con 0x7fcdac0f69d0 2024-01-18T06:49:03.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:49:03.917+0000 7fcdb1d57700 1 -- 172.21.15.47:0/438697654 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fcd9802ff60 msgr2=0x7fcd98032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:03.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:49:03.917+0000 7fcdb1d57700 1 --2- 172.21.15.47:0/438697654 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fcd9802ff60 0x7fcd98032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fcd940096a0 tx=0x7fcd94006b40).stop 2024-01-18T06:49:03.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:49:03.917+0000 7fcdb1d57700 1 -- 172.21.15.47:0/438697654 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcdac0f69d0 msgr2=0x7fcdac0fd390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:03.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:49:03.917+0000 7fcdb1d57700 1 --2- 172.21.15.47:0/438697654 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcdac0f69d0 0x7fcdac0fd390 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fcd9c013510 tx=0x7fcd9c00b040).stop 2024-01-18T06:49:03.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:49:03.917+0000 7fcdb1d57700 1 -- 172.21.15.47:0/438697654 shutdown_connections 2024-01-18T06:49:03.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:49:03.917+0000 7fcdb1d57700 1 --2- 172.21.15.47:0/438697654 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fcd9802ff60 0x7fcd98032420 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:03.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:49:03.917+0000 7fcdb1d57700 1 --2- 172.21.15.47:0/438697654 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcdac0f69d0 0x7fcdac0fd390 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:03.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:49:03.917+0000 7fcdb1d57700 1 -- 172.21.15.47:0/438697654 >> 172.21.15.47:0/438697654 conn(0x7fcdac0f02e0 msgr2=0x7fcdac0f4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:49:03.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:49:03.917+0000 7fcdb1d57700 1 -- 172.21.15.47:0/438697654 shutdown_connections 2024-01-18T06:49:03.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-01-18T06:49:03.917+0000 7fcdb1d57700 1 -- 172.21.15.47:0/438697654 wait complete. 2024-01-18T06:49:03.919 INFO:teuthology.orchestra.run.smithi047.stderr:adjusted tunables profile to default 2024-01-18T06:49:03.963 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:03 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:49:01.976350+0000 mgr.smithi047.fuhzzr (mgr.14162) 15 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:49:03.963 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:03 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:49:03.189389+0000 mon.smithi047 (mon.0) 117 : audit [INF] from='client.? 172.21.15.47:0/438697654' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-01-18T06:49:04.483 INFO:tasks.cephadm:Adding mon.smithi047 on smithi047 2024-01-18T06:49:04.483 INFO:tasks.cephadm:Adding mon.smithi194 on smithi194 2024-01-18T06:49:04.484 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph orch apply mon '2;smithi047:172.21.15.47=smithi047;smithi194:172.21.15.194=smithi194' 2024-01-18T06:49:05.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:04 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:49:03.916567+0000 mon.smithi047 (mon.0) 118 : audit [INF] from='client.? 172.21.15.47:0/438697654' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-01-18T06:49:05.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:04 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:49:03.916621+0000 mon.smithi047 (mon.0) 119 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-01-18T06:49:05.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:04 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:49:03.976645+0000 mgr.smithi047.fuhzzr (mgr.14162) 16 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:49:05.677 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:05.676+0000 7efe453bb700 1 -- 172.21.15.194:0/1446586978 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe400f51a0 msgr2=0x7efe400f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:05.677 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:05.676+0000 7efe453bb700 1 --2- 172.21.15.194:0/1446586978 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe400f51a0 0x7efe400f55c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7efe30009fb0 tx=0x7efe30009b30).stop 2024-01-18T06:49:05.677 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:05.676+0000 7efe453bb700 1 -- 172.21.15.194:0/1446586978 shutdown_connections 2024-01-18T06:49:05.677 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:05.676+0000 7efe453bb700 1 --2- 172.21.15.194:0/1446586978 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe400f51a0 0x7efe400f55c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:05.677 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:05.676+0000 7efe453bb700 1 -- 172.21.15.194:0/1446586978 >> 172.21.15.194:0/1446586978 conn(0x7efe400f09a0 msgr2=0x7efe400f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:49:05.677 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:05.676+0000 7efe453bb700 1 -- 172.21.15.194:0/1446586978 shutdown_connections 2024-01-18T06:49:05.677 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:05.676+0000 7efe453bb700 1 -- 172.21.15.194:0/1446586978 wait complete. 2024-01-18T06:49:05.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:05.677+0000 7efe453bb700 1 Processor -- start 2024-01-18T06:49:05.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:05.677+0000 7efe453bb700 1 -- start start 2024-01-18T06:49:05.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:05.677+0000 7efe453bb700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe400f51a0 0x7efe401053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:49:05.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:05.677+0000 7efe453bb700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe401058a0 con 0x7efe400f51a0 2024-01-18T06:49:05.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:05.677+0000 7efe3ffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe400f51a0 0x7efe401053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:49:05.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:05.677+0000 7efe3ffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe400f51a0 0x7efe401053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:52880/0 (socket says 172.21.15.194:52880) 2024-01-18T06:49:05.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:05.677+0000 7efe3ffff700 1 -- 172.21.15.194:0/2278904313 learned_addr learned my addr 172.21.15.194:0/2278904313 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:49:05.681 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:05.678+0000 7efe3ffff700 1 -- 172.21.15.194:0/2278904313 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe30005040 con 0x7efe400f51a0 2024-01-18T06:49:05.681 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:05.678+0000 7efe3ffff700 1 --2- 172.21.15.194:0/2278904313 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe400f51a0 0x7efe401053b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7efe30009830 tx=0x7efe3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:49:05.681 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:05.678+0000 7efe3cff9700 1 -- 172.21.15.194:0/2278904313 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe30009320 con 0x7efe400f51a0 2024-01-18T06:49:05.681 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:05.678+0000 7efe453bb700 1 -- 172.21.15.194:0/2278904313 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe40105aa0 con 0x7efe400f51a0 2024-01-18T06:49:05.681 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:05.678+0000 7efe3cff9700 1 -- 172.21.15.194:0/2278904313 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efe300136c0 con 0x7efe400f51a0 2024-01-18T06:49:05.681 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:05.678+0000 7efe3cff9700 1 -- 172.21.15.194:0/2278904313 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe3001ee90 con 0x7efe400f51a0 2024-01-18T06:49:05.681 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:05.678+0000 7efe453bb700 1 -- 172.21.15.194:0/2278904313 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe40105f30 con 0x7efe400f51a0 2024-01-18T06:49:05.681 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:05.679+0000 7efe3cff9700 1 -- 172.21.15.194:0/2278904313 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efe3001e420 con 0x7efe400f51a0 2024-01-18T06:49:05.681 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:05.679+0000 7efe453bb700 1 -- 172.21.15.194:0/2278904313 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe40045ad0 con 0x7efe400f51a0 2024-01-18T06:49:05.681 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:05.679+0000 7efe3cff9700 1 --2- 172.21.15.194:0/2278904313 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7efe2802ff10 0x7efe280323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:49:05.682 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:05.679+0000 7efe3cff9700 1 -- 172.21.15.194:0/2278904313 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efe30041050 con 0x7efe400f51a0 2024-01-18T06:49:05.682 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:05.679+0000 7efe3f7fe700 1 --2- 172.21.15.194:0/2278904313 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7efe2802ff10 0x7efe280323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:49:05.682 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:05.680+0000 7efe3f7fe700 1 --2- 172.21.15.194:0/2278904313 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7efe2802ff10 0x7efe280323d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7efe340096a0 tx=0x7efe34006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:49:05.684 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:05.684+0000 7efe3cff9700 1 -- 172.21.15.194:0/2278904313 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efe3006d2b0 con 0x7efe400f51a0 2024-01-18T06:49:05.860 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:05.859+0000 7efe453bb700 1 -- 172.21.15.194:0/2278904313 --> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi047:172.21.15.47=smithi047;smithi194:172.21.15.194=smithi194", "target": ["mon-mgr", ""]}) v1 -- 0x7efe400f1c20 con 0x7efe2802ff10 2024-01-18T06:49:05.865 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:05.866+0000 7efe3cff9700 1 -- 172.21.15.194:0/2278904313 <== mgr.14162 v2:172.21.15.47:6800/1933549322 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7efe400f1c20 con 0x7efe2802ff10 2024-01-18T06:49:05.865 INFO:teuthology.orchestra.run.smithi194.stdout:Scheduled mon update... 2024-01-18T06:49:05.867 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:05.867+0000 7efe453bb700 1 -- 172.21.15.194:0/2278904313 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7efe2802ff10 msgr2=0x7efe280323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:05.867 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:05.867+0000 7efe453bb700 1 --2- 172.21.15.194:0/2278904313 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7efe2802ff10 0x7efe280323d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7efe340096a0 tx=0x7efe34006b40).stop 2024-01-18T06:49:05.867 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:05.867+0000 7efe453bb700 1 -- 172.21.15.194:0/2278904313 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe400f51a0 msgr2=0x7efe401053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:05.868 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:05.867+0000 7efe453bb700 1 --2- 172.21.15.194:0/2278904313 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe400f51a0 0x7efe401053b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7efe30009830 tx=0x7efe3000b040).stop 2024-01-18T06:49:05.868 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:05.867+0000 7efe453bb700 1 -- 172.21.15.194:0/2278904313 shutdown_connections 2024-01-18T06:49:05.868 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:05.867+0000 7efe453bb700 1 --2- 172.21.15.194:0/2278904313 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7efe2802ff10 0x7efe280323d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:05.868 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:05.867+0000 7efe453bb700 1 --2- 172.21.15.194:0/2278904313 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe400f51a0 0x7efe401053b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:05.868 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:05.867+0000 7efe453bb700 1 -- 172.21.15.194:0/2278904313 >> 172.21.15.194:0/2278904313 conn(0x7efe400f09a0 msgr2=0x7efe400f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:49:05.868 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:05.867+0000 7efe453bb700 1 -- 172.21.15.194:0/2278904313 shutdown_connections 2024-01-18T06:49:05.868 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:05.867+0000 7efe453bb700 1 -- 172.21.15.194:0/2278904313 wait complete. 2024-01-18T06:49:06.270 DEBUG:teuthology.orchestra.run.smithi194:mon.smithi194> sudo journalctl -f -n 0 -u ceph-653e4012-b5cd-11ee-95ae-87774f69a715@mon.smithi194.service 2024-01-18T06:49:06.273 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:49:06.273 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:49:06.301 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:-- Logs begin at Thu 2024-01-18 06:37:28 UTC. -- 2024-01-18T06:49:07.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:06 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:49:05.862379+0000 mgr.smithi047.fuhzzr (mgr.14162) 17 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi047:172.21.15.47=smithi047;smithi194:172.21.15.194=smithi194", "target": ["mon-mgr", ""]}]: dispatch 2024-01-18T06:49:07.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:06 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cephadm 2024-01-18T06:49:05.863814+0000 mgr.smithi047.fuhzzr (mgr.14162) 18 : cephadm [INF] Saving service mon spec with placement smithi047:172.21.15.47=smithi047;smithi194:172.21.15.194=smithi194;count:2 2024-01-18T06:49:07.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:06 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:49:05.866353+0000 mon.smithi047 (mon.0) 120 : audit [INF] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' 2024-01-18T06:49:07.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:06 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:49:05.976899+0000 mgr.smithi047.fuhzzr (mgr.14162) 19 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:49:07.436 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:07.436+0000 7f3d00779700 1 -- 172.21.15.194:0/1162455287 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3cf80f5410 msgr2=0x7f3cf80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:07.437 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:07.436+0000 7f3d00779700 1 --2- 172.21.15.194:0/1162455287 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3cf80f5410 0x7f3cf80f5830 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f3cec009fb0 tx=0x7f3cec009b30).stop 2024-01-18T06:49:07.437 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:07.436+0000 7f3d00779700 1 -- 172.21.15.194:0/1162455287 shutdown_connections 2024-01-18T06:49:07.437 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:07.436+0000 7f3d00779700 1 --2- 172.21.15.194:0/1162455287 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3cf80f5410 0x7f3cf80f5830 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:07.437 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:07.436+0000 7f3d00779700 1 -- 172.21.15.194:0/1162455287 >> 172.21.15.194:0/1162455287 conn(0x7f3cf80f0b70 msgr2=0x7f3cf80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:49:07.437 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:07.436+0000 7f3d00779700 1 -- 172.21.15.194:0/1162455287 shutdown_connections 2024-01-18T06:49:07.437 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:07.436+0000 7f3d00779700 1 -- 172.21.15.194:0/1162455287 wait complete. 2024-01-18T06:49:07.439 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:07.436+0000 7f3d00779700 1 Processor -- start 2024-01-18T06:49:07.439 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:07.436+0000 7f3d00779700 1 -- start start 2024-01-18T06:49:07.439 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:07.437+0000 7f3d00779700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3cf80f5410 0x7f3cf8105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:49:07.439 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:07.437+0000 7f3d00779700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf8105b20 con 0x7f3cf80f5410 2024-01-18T06:49:07.439 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:07.437+0000 7f3cfe515700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3cf80f5410 0x7f3cf8105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:49:07.439 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:07.437+0000 7f3cfe515700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3cf80f5410 0x7f3cf8105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:37562/0 (socket says 172.21.15.194:37562) 2024-01-18T06:49:07.439 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:07.437+0000 7f3cfe515700 1 -- 172.21.15.194:0/198599961 learned_addr learned my addr 172.21.15.194:0/198599961 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:49:07.440 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:07.437+0000 7f3cfe515700 1 -- 172.21.15.194:0/198599961 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cec005040 con 0x7f3cf80f5410 2024-01-18T06:49:07.440 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:07.437+0000 7f3cfe515700 1 --2- 172.21.15.194:0/198599961 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3cf80f5410 0x7f3cf8105630 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f3cec013510 tx=0x7f3cec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:49:07.440 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:07.437+0000 7f3ceaffd700 1 -- 172.21.15.194:0/198599961 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3cec009320 con 0x7f3cf80f5410 2024-01-18T06:49:07.440 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:07.437+0000 7f3ceaffd700 1 -- 172.21.15.194:0/198599961 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3cec013f00 con 0x7f3cf80f5410 2024-01-18T06:49:07.440 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:07.437+0000 7f3d00779700 1 -- 172.21.15.194:0/198599961 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cf8105d20 con 0x7f3cf80f5410 2024-01-18T06:49:07.440 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:07.437+0000 7f3ceaffd700 1 -- 172.21.15.194:0/198599961 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3cec014d90 con 0x7f3cf80f5410 2024-01-18T06:49:07.440 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:07.438+0000 7f3d00779700 1 -- 172.21.15.194:0/198599961 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cf81061b0 con 0x7f3cf80f5410 2024-01-18T06:49:07.440 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:07.438+0000 7f3ceaffd700 1 -- 172.21.15.194:0/198599961 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3cec0095f0 con 0x7f3cf80f5410 2024-01-18T06:49:07.440 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:07.438+0000 7f3d00779700 1 -- 172.21.15.194:0/198599961 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cf8045ad0 con 0x7f3cf80f5410 2024-01-18T06:49:07.440 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:07.438+0000 7f3ceaffd700 1 --2- 172.21.15.194:0/198599961 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3ce402ff60 0x7f3ce4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:49:07.440 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:07.438+0000 7f3ceaffd700 1 -- 172.21.15.194:0/198599961 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3cec041080 con 0x7f3cf80f5410 2024-01-18T06:49:07.440 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:07.439+0000 7f3cfdd14700 1 --2- 172.21.15.194:0/198599961 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3ce402ff60 0x7f3ce4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:49:07.441 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:07.439+0000 7f3cfdd14700 1 --2- 172.21.15.194:0/198599961 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3ce402ff60 0x7f3ce4032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f3cf40096a0 tx=0x7f3cf4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:49:07.443 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:07.444+0000 7f3ceaffd700 1 -- 172.21.15.194:0/198599961 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3cec01d7f0 con 0x7f3cf80f5410 2024-01-18T06:49:07.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:07.648+0000 7f3d00779700 1 -- 172.21.15.194:0/198599961 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3cf80fae60 con 0x7f3cf80f5410 2024-01-18T06:49:07.650 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:07.649+0000 7f3ceaffd700 1 -- 172.21.15.194:0/198599961 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3cec010230 con 0x7f3cf80f5410 2024-01-18T06:49:07.650 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:49:07.650 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:49:07.652 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:07.651+0000 7f3d00779700 1 -- 172.21.15.194:0/198599961 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3ce402ff60 msgr2=0x7f3ce4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:07.652 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:07.651+0000 7f3d00779700 1 --2- 172.21.15.194:0/198599961 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3ce402ff60 0x7f3ce4032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f3cf40096a0 tx=0x7f3cf4006b40).stop 2024-01-18T06:49:07.652 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:07.651+0000 7f3d00779700 1 -- 172.21.15.194:0/198599961 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3cf80f5410 msgr2=0x7f3cf8105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:07.652 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:07.651+0000 7f3d00779700 1 --2- 172.21.15.194:0/198599961 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3cf80f5410 0x7f3cf8105630 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f3cec013510 tx=0x7f3cec00b040).stop 2024-01-18T06:49:07.652 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:07.651+0000 7f3d00779700 1 -- 172.21.15.194:0/198599961 shutdown_connections 2024-01-18T06:49:07.652 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:07.651+0000 7f3d00779700 1 --2- 172.21.15.194:0/198599961 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3ce402ff60 0x7f3ce4032420 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:07.652 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:07.651+0000 7f3d00779700 1 --2- 172.21.15.194:0/198599961 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3cf80f5410 0x7f3cf8105630 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:07.652 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:07.651+0000 7f3d00779700 1 -- 172.21.15.194:0/198599961 >> 172.21.15.194:0/198599961 conn(0x7f3cf80f0b70 msgr2=0x7f3cf80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:49:07.652 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:07.651+0000 7f3d00779700 1 -- 172.21.15.194:0/198599961 shutdown_connections 2024-01-18T06:49:07.652 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:07.651+0000 7f3d00779700 1 -- 172.21.15.194:0/198599961 wait complete. 2024-01-18T06:49:07.653 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:49:08.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:07 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:49:07.651108+0000 mon.smithi047 (mon.0) 121 : audit [DBG] from='client.? 172.21.15.194:0/198599961' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:49:09.028 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:49:09.028 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:49:09.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:08 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:49:07.977308+0000 mgr.smithi047.fuhzzr (mgr.14162) 20 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:49:10.301 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:10.301+0000 7fb59613d700 1 -- 172.21.15.194:0/965925827 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb5900f5410 msgr2=0x7fb5900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:10.301 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:10.301+0000 7fb59613d700 1 --2- 172.21.15.194:0/965925827 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb5900f5410 0x7fb5900f5830 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fb580009fb0 tx=0x7fb580009b30).stop 2024-01-18T06:49:10.301 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:10.301+0000 7fb59613d700 1 -- 172.21.15.194:0/965925827 shutdown_connections 2024-01-18T06:49:10.301 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:10.301+0000 7fb59613d700 1 --2- 172.21.15.194:0/965925827 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb5900f5410 0x7fb5900f5830 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:10.301 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:10.301+0000 7fb59613d700 1 -- 172.21.15.194:0/965925827 >> 172.21.15.194:0/965925827 conn(0x7fb5900f0b70 msgr2=0x7fb5900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:49:10.301 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:10.301+0000 7fb59613d700 1 -- 172.21.15.194:0/965925827 shutdown_connections 2024-01-18T06:49:10.301 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:10.301+0000 7fb59613d700 1 -- 172.21.15.194:0/965925827 wait complete. 2024-01-18T06:49:10.301 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:10.301+0000 7fb59613d700 1 Processor -- start 2024-01-18T06:49:10.303 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:10.302+0000 7fb59613d700 1 -- start start 2024-01-18T06:49:10.303 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:10.302+0000 7fb59613d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb5900f5410 0x7fb590105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:49:10.303 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:10.302+0000 7fb59613d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb590105b20 con 0x7fb5900f5410 2024-01-18T06:49:10.303 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:10.302+0000 7fb58f7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb5900f5410 0x7fb590105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:49:10.303 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:10.302+0000 7fb58f7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb5900f5410 0x7fb590105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:37582/0 (socket says 172.21.15.194:37582) 2024-01-18T06:49:10.303 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:10.302+0000 7fb58f7fe700 1 -- 172.21.15.194:0/2466093803 learned_addr learned my addr 172.21.15.194:0/2466093803 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:49:10.303 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:10.302+0000 7fb58f7fe700 1 -- 172.21.15.194:0/2466093803 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb580005040 con 0x7fb5900f5410 2024-01-18T06:49:10.303 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:10.303+0000 7fb58f7fe700 1 --2- 172.21.15.194:0/2466093803 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb5900f5410 0x7fb590105630 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fb580014000 tx=0x7fb58000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:49:10.303 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:10.303+0000 7fb577fff700 1 -- 172.21.15.194:0/2466093803 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5800095f0 con 0x7fb5900f5410 2024-01-18T06:49:10.303 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:10.303+0000 7fb59613d700 1 -- 172.21.15.194:0/2466093803 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb590105d20 con 0x7fb5900f5410 2024-01-18T06:49:10.303 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:10.303+0000 7fb577fff700 1 -- 172.21.15.194:0/2466093803 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb580007e70 con 0x7fb5900f5410 2024-01-18T06:49:10.303 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:10.303+0000 7fb577fff700 1 -- 172.21.15.194:0/2466093803 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb580014ea0 con 0x7fb5900f5410 2024-01-18T06:49:10.303 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:10.303+0000 7fb59613d700 1 -- 172.21.15.194:0/2466093803 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5901061b0 con 0x7fb5900f5410 2024-01-18T06:49:10.304 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:10.304+0000 7fb577fff700 1 -- 172.21.15.194:0/2466093803 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb580025070 con 0x7fb5900f5410 2024-01-18T06:49:10.304 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:10.304+0000 7fb59613d700 1 -- 172.21.15.194:0/2466093803 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb590045ad0 con 0x7fb5900f5410 2024-01-18T06:49:10.304 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:10.304+0000 7fb577fff700 1 --2- 172.21.15.194:0/2466093803 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb57802ff60 0x7fb578032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:49:10.304 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:10.304+0000 7fb577fff700 1 -- 172.21.15.194:0/2466093803 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb5800409c0 con 0x7fb5900f5410 2024-01-18T06:49:10.305 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:10.304+0000 7fb58effd700 1 --2- 172.21.15.194:0/2466093803 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb57802ff60 0x7fb578032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:49:10.305 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:10.304+0000 7fb58effd700 1 --2- 172.21.15.194:0/2466093803 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb57802ff60 0x7fb578032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fb5840096a0 tx=0x7fb584006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:49:10.308 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:10.308+0000 7fb577fff700 1 -- 172.21.15.194:0/2466093803 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb5800079c0 con 0x7fb5900f5410 2024-01-18T06:49:10.514 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:10.514+0000 7fb59613d700 1 -- 172.21.15.194:0/2466093803 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb5900fae60 con 0x7fb5900f5410 2024-01-18T06:49:10.514 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:10.515+0000 7fb577fff700 1 -- 172.21.15.194:0/2466093803 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb58000f300 con 0x7fb5900f5410 2024-01-18T06:49:10.515 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:49:10.515 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:49:10.516 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:10.516+0000 7fb59613d700 1 -- 172.21.15.194:0/2466093803 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb57802ff60 msgr2=0x7fb578032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:10.516 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:10.516+0000 7fb59613d700 1 --2- 172.21.15.194:0/2466093803 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb57802ff60 0x7fb578032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fb5840096a0 tx=0x7fb584006b40).stop 2024-01-18T06:49:10.516 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:10.516+0000 7fb59613d700 1 -- 172.21.15.194:0/2466093803 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb5900f5410 msgr2=0x7fb590105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:10.516 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:10.516+0000 7fb59613d700 1 --2- 172.21.15.194:0/2466093803 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb5900f5410 0x7fb590105630 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fb580014000 tx=0x7fb58000b040).stop 2024-01-18T06:49:10.516 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:10.516+0000 7fb59613d700 1 -- 172.21.15.194:0/2466093803 shutdown_connections 2024-01-18T06:49:10.516 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:10.516+0000 7fb59613d700 1 --2- 172.21.15.194:0/2466093803 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb57802ff60 0x7fb578032420 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:10.517 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:10.516+0000 7fb59613d700 1 --2- 172.21.15.194:0/2466093803 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb5900f5410 0x7fb590105630 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:10.517 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:10.516+0000 7fb59613d700 1 -- 172.21.15.194:0/2466093803 >> 172.21.15.194:0/2466093803 conn(0x7fb5900f0b70 msgr2=0x7fb5900f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:49:10.517 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:10.517+0000 7fb59613d700 1 -- 172.21.15.194:0/2466093803 shutdown_connections 2024-01-18T06:49:10.517 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:10.517+0000 7fb59613d700 1 -- 172.21.15.194:0/2466093803 wait complete. 2024-01-18T06:49:10.517 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:49:11.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:11 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:49:09.977759+0000 mgr.smithi047.fuhzzr (mgr.14162) 21 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:49:11.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:11 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:49:10.515982+0000 mon.smithi047 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.194:0/2466093803' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:49:11.897 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:49:11.897 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:49:13.074 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:13.073+0000 7f3f31247700 1 -- 172.21.15.194:0/2174951481 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3f2c0f6510 msgr2=0x7f3f2c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:13.074 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:13.073+0000 7f3f31247700 1 --2- 172.21.15.194:0/2174951481 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3f2c0f6510 0x7f3f2c0f6930 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f3f20009fb0 tx=0x7f3f20009b30).stop 2024-01-18T06:49:13.074 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:13.073+0000 7f3f31247700 1 -- 172.21.15.194:0/2174951481 shutdown_connections 2024-01-18T06:49:13.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:13.073+0000 7f3f31247700 1 --2- 172.21.15.194:0/2174951481 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3f2c0f6510 0x7f3f2c0f6930 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:13.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:13.073+0000 7f3f31247700 1 -- 172.21.15.194:0/2174951481 >> 172.21.15.194:0/2174951481 conn(0x7f3f2c0f1c70 msgr2=0x7f3f2c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:49:13.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:13.073+0000 7f3f31247700 1 -- 172.21.15.194:0/2174951481 shutdown_connections 2024-01-18T06:49:13.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:13.073+0000 7f3f31247700 1 -- 172.21.15.194:0/2174951481 wait complete. 2024-01-18T06:49:13.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:13.073+0000 7f3f31247700 1 Processor -- start 2024-01-18T06:49:13.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:13.073+0000 7f3f31247700 1 -- start start 2024-01-18T06:49:13.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:13.074+0000 7f3f31247700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3f2c0f6510 0x7f3f2c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:49:13.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:13.074+0000 7f3f31247700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2c069e20 con 0x7f3f2c0f6510 2024-01-18T06:49:13.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:13.074+0000 7f3f2ad9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3f2c0f6510 0x7f3f2c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:49:13.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:13.074+0000 7f3f2ad9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3f2c0f6510 0x7f3f2c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:37598/0 (socket says 172.21.15.194:37598) 2024-01-18T06:49:13.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:13.074+0000 7f3f2ad9d700 1 -- 172.21.15.194:0/4254369508 learned_addr learned my addr 172.21.15.194:0/4254369508 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:49:13.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:13.074+0000 7f3f2ad9d700 1 -- 172.21.15.194:0/4254369508 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f20005040 con 0x7f3f2c0f6510 2024-01-18T06:49:13.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:13.074+0000 7f3f2ad9d700 1 --2- 172.21.15.194:0/4254369508 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3f2c0f6510 0x7f3f2c06b9a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f3f20014000 tx=0x7f3f2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:49:13.076 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:13.075+0000 7f3f137fe700 1 -- 172.21.15.194:0/4254369508 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f200095f0 con 0x7f3f2c0f6510 2024-01-18T06:49:13.076 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:13.075+0000 7f3f31247700 1 -- 172.21.15.194:0/4254369508 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f2c069fc0 con 0x7f3f2c0f6510 2024-01-18T06:49:13.076 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:13.075+0000 7f3f137fe700 1 -- 172.21.15.194:0/4254369508 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3f20007e70 con 0x7f3f2c0f6510 2024-01-18T06:49:13.076 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:13.075+0000 7f3f137fe700 1 -- 172.21.15.194:0/4254369508 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f20014ea0 con 0x7f3f2c0f6510 2024-01-18T06:49:13.076 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:13.075+0000 7f3f31247700 1 -- 172.21.15.194:0/4254369508 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f2c06a450 con 0x7f3f2c0f6510 2024-01-18T06:49:13.077 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:13.075+0000 7f3f137fe700 1 -- 172.21.15.194:0/4254369508 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3f20025070 con 0x7f3f2c0f6510 2024-01-18T06:49:13.077 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:13.075+0000 7f3f31247700 1 -- 172.21.15.194:0/4254369508 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f2c059100 con 0x7f3f2c0f6510 2024-01-18T06:49:13.077 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:13.076+0000 7f3f137fe700 1 --2- 172.21.15.194:0/4254369508 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3f1402ffb0 0x7f3f14032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:49:13.077 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:13.076+0000 7f3f137fe700 1 -- 172.21.15.194:0/4254369508 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3f200409f0 con 0x7f3f2c0f6510 2024-01-18T06:49:13.077 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:13.076+0000 7f3f2a59c700 1 --2- 172.21.15.194:0/4254369508 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3f1402ffb0 0x7f3f14032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:49:13.077 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:13.076+0000 7f3f2a59c700 1 --2- 172.21.15.194:0/4254369508 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3f1402ffb0 0x7f3f14032470 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f3f1c0096a0 tx=0x7f3f1c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:49:13.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:13.080+0000 7f3f137fe700 1 -- 172.21.15.194:0/4254369508 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3f200079c0 con 0x7f3f2c0f6510 2024-01-18T06:49:13.283 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:13.283+0000 7f3f31247700 1 -- 172.21.15.194:0/4254369508 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3f2c0fbf60 con 0x7f3f2c0f6510 2024-01-18T06:49:13.283 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:13.284+0000 7f3f137fe700 1 -- 172.21.15.194:0/4254369508 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3f2000f300 con 0x7f3f2c0f6510 2024-01-18T06:49:13.283 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:49:13.283 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:49:13.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:13.285+0000 7f3f31247700 1 -- 172.21.15.194:0/4254369508 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3f1402ffb0 msgr2=0x7f3f14032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:13.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:13.285+0000 7f3f31247700 1 --2- 172.21.15.194:0/4254369508 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3f1402ffb0 0x7f3f14032470 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f3f1c0096a0 tx=0x7f3f1c006b40).stop 2024-01-18T06:49:13.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:13.285+0000 7f3f31247700 1 -- 172.21.15.194:0/4254369508 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3f2c0f6510 msgr2=0x7f3f2c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:13.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:13.285+0000 7f3f31247700 1 --2- 172.21.15.194:0/4254369508 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3f2c0f6510 0x7f3f2c06b9a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f3f20014000 tx=0x7f3f2000b040).stop 2024-01-18T06:49:13.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:13.285+0000 7f3f31247700 1 -- 172.21.15.194:0/4254369508 shutdown_connections 2024-01-18T06:49:13.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:13.285+0000 7f3f31247700 1 --2- 172.21.15.194:0/4254369508 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3f1402ffb0 0x7f3f14032470 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:13.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:13.285+0000 7f3f31247700 1 --2- 172.21.15.194:0/4254369508 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3f2c0f6510 0x7f3f2c06b9a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:13.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:13.285+0000 7f3f31247700 1 -- 172.21.15.194:0/4254369508 >> 172.21.15.194:0/4254369508 conn(0x7f3f2c0f1c70 msgr2=0x7f3f2c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:49:13.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:13.285+0000 7f3f31247700 1 -- 172.21.15.194:0/4254369508 shutdown_connections 2024-01-18T06:49:13.286 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:13.285+0000 7f3f31247700 1 -- 172.21.15.194:0/4254369508 wait complete. 2024-01-18T06:49:13.286 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:49:13.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:13 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:49:11.978125+0000 mgr.smithi047.fuhzzr (mgr.14162) 22 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:49:14.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:14 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:49:13.284696+0000 mon.smithi047 (mon.0) 123 : audit [DBG] from='client.? 172.21.15.194:0/4254369508' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:49:14.689 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:49:14.689 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:49:15.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:15 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:49:13.978592+0000 mgr.smithi047.fuhzzr (mgr.14162) 23 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:49:15.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:15.938+0000 7fb5d8cc7700 1 -- 172.21.15.194:0/176842079 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb5d40f6510 msgr2=0x7fb5d40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:15.940 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:15.938+0000 7fb5d8cc7700 1 --2- 172.21.15.194:0/176842079 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb5d40f6510 0x7fb5d40f6930 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fb5c4009fb0 tx=0x7fb5c4009b30).stop 2024-01-18T06:49:15.940 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:15.938+0000 7fb5d8cc7700 1 -- 172.21.15.194:0/176842079 shutdown_connections 2024-01-18T06:49:15.940 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:15.938+0000 7fb5d8cc7700 1 --2- 172.21.15.194:0/176842079 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb5d40f6510 0x7fb5d40f6930 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:15.940 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:15.938+0000 7fb5d8cc7700 1 -- 172.21.15.194:0/176842079 >> 172.21.15.194:0/176842079 conn(0x7fb5d40f1c70 msgr2=0x7fb5d40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:49:15.940 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:15.938+0000 7fb5d8cc7700 1 -- 172.21.15.194:0/176842079 shutdown_connections 2024-01-18T06:49:15.940 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:15.938+0000 7fb5d8cc7700 1 -- 172.21.15.194:0/176842079 wait complete. 2024-01-18T06:49:15.940 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:15.939+0000 7fb5d8cc7700 1 Processor -- start 2024-01-18T06:49:15.940 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:15.939+0000 7fb5d8cc7700 1 -- start start 2024-01-18T06:49:15.940 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:15.939+0000 7fb5d8cc7700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb5d40f6510 0x7fb5d4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:49:15.940 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:15.939+0000 7fb5d8cc7700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d4106c20 con 0x7fb5d40f6510 2024-01-18T06:49:15.940 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:15.939+0000 7fb5d259c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb5d40f6510 0x7fb5d4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:49:15.940 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:15.939+0000 7fb5d259c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb5d40f6510 0x7fb5d4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:37620/0 (socket says 172.21.15.194:37620) 2024-01-18T06:49:15.940 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:15.939+0000 7fb5d259c700 1 -- 172.21.15.194:0/3911684661 learned_addr learned my addr 172.21.15.194:0/3911684661 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:49:15.941 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:15.940+0000 7fb5d259c700 1 -- 172.21.15.194:0/3911684661 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c4005040 con 0x7fb5d40f6510 2024-01-18T06:49:15.941 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:15.940+0000 7fb5d259c700 1 --2- 172.21.15.194:0/3911684661 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb5d40f6510 0x7fb5d4106730 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fb5c4013510 tx=0x7fb5c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:49:15.941 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:15.940+0000 7fb5c2ffd700 1 -- 172.21.15.194:0/3911684661 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5c4009320 con 0x7fb5d40f6510 2024-01-18T06:49:15.941 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:15.940+0000 7fb5c2ffd700 1 -- 172.21.15.194:0/3911684661 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb5c4013f00 con 0x7fb5d40f6510 2024-01-18T06:49:15.941 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:15.940+0000 7fb5d8cc7700 1 -- 172.21.15.194:0/3911684661 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5d4106e20 con 0x7fb5d40f6510 2024-01-18T06:49:15.941 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:15.940+0000 7fb5c2ffd700 1 -- 172.21.15.194:0/3911684661 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5c4014d90 con 0x7fb5d40f6510 2024-01-18T06:49:15.941 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:15.940+0000 7fb5d8cc7700 1 -- 172.21.15.194:0/3911684661 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5d41072b0 con 0x7fb5d40f6510 2024-01-18T06:49:15.942 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:15.941+0000 7fb5c2ffd700 1 -- 172.21.15.194:0/3911684661 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb5c40095f0 con 0x7fb5d40f6510 2024-01-18T06:49:15.942 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:15.941+0000 7fb5d8cc7700 1 -- 172.21.15.194:0/3911684661 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5d4045ad0 con 0x7fb5d40f6510 2024-01-18T06:49:15.942 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:15.941+0000 7fb5c2ffd700 1 --2- 172.21.15.194:0/3911684661 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb5bc02ff60 0x7fb5bc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:49:15.942 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:15.941+0000 7fb5c2ffd700 1 -- 172.21.15.194:0/3911684661 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb5c4041000 con 0x7fb5d40f6510 2024-01-18T06:49:15.942 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:15.941+0000 7fb5d1d9b700 1 --2- 172.21.15.194:0/3911684661 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb5bc02ff60 0x7fb5bc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:49:15.942 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:15.941+0000 7fb5d1d9b700 1 --2- 172.21.15.194:0/3911684661 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb5bc02ff60 0x7fb5bc032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fb5c80096a0 tx=0x7fb5c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:49:15.947 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:15.946+0000 7fb5c2ffd700 1 -- 172.21.15.194:0/3911684661 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb5c4014420 con 0x7fb5d40f6510 2024-01-18T06:49:16.150 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:16.151+0000 7fb5d8cc7700 1 -- 172.21.15.194:0/3911684661 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb5d40fbf60 con 0x7fb5d40f6510 2024-01-18T06:49:16.151 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:16.152+0000 7fb5c2ffd700 1 -- 172.21.15.194:0/3911684661 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb5c4010230 con 0x7fb5d40f6510 2024-01-18T06:49:16.151 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:49:16.151 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:49:16.153 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:16.153+0000 7fb5d8cc7700 1 -- 172.21.15.194:0/3911684661 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb5bc02ff60 msgr2=0x7fb5bc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:16.153 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:16.153+0000 7fb5d8cc7700 1 --2- 172.21.15.194:0/3911684661 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb5bc02ff60 0x7fb5bc032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fb5c80096a0 tx=0x7fb5c8006b40).stop 2024-01-18T06:49:16.153 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:16.153+0000 7fb5d8cc7700 1 -- 172.21.15.194:0/3911684661 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb5d40f6510 msgr2=0x7fb5d4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:16.153 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:16.153+0000 7fb5d8cc7700 1 --2- 172.21.15.194:0/3911684661 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb5d40f6510 0x7fb5d4106730 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fb5c4013510 tx=0x7fb5c400b040).stop 2024-01-18T06:49:16.153 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:16.153+0000 7fb5d8cc7700 1 -- 172.21.15.194:0/3911684661 shutdown_connections 2024-01-18T06:49:16.153 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:16.153+0000 7fb5d8cc7700 1 --2- 172.21.15.194:0/3911684661 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb5bc02ff60 0x7fb5bc032420 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:16.153 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:16.153+0000 7fb5d8cc7700 1 --2- 172.21.15.194:0/3911684661 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb5d40f6510 0x7fb5d4106730 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:16.153 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:16.153+0000 7fb5d8cc7700 1 -- 172.21.15.194:0/3911684661 >> 172.21.15.194:0/3911684661 conn(0x7fb5d40f1c70 msgr2=0x7fb5d40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:49:16.153 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:16.153+0000 7fb5d8cc7700 1 -- 172.21.15.194:0/3911684661 shutdown_connections 2024-01-18T06:49:16.154 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:16.153+0000 7fb5d8cc7700 1 -- 172.21.15.194:0/3911684661 wait complete. 2024-01-18T06:49:16.154 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:49:17.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:17 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:49:15.978809+0000 mgr.smithi047.fuhzzr (mgr.14162) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:49:17.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:17 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:49:16.152756+0000 mon.smithi047 (mon.0) 124 : audit [DBG] from='client.? 172.21.15.194:0/3911684661' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:49:17.546 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:49:17.546 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:49:18.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:18.738+0000 7f005caba700 1 -- 172.21.15.194:0/3703874879 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f00580f6430 msgr2=0x7f00580f6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:18.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:18.738+0000 7f005caba700 1 --2- 172.21.15.194:0/3703874879 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f00580f6430 0x7f00580f6850 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f0040009fb0 tx=0x7f0040009b30).stop 2024-01-18T06:49:18.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:18.739+0000 7f005caba700 1 -- 172.21.15.194:0/3703874879 shutdown_connections 2024-01-18T06:49:18.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:18.739+0000 7f005caba700 1 --2- 172.21.15.194:0/3703874879 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f00580f6430 0x7f00580f6850 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:18.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:18.739+0000 7f005caba700 1 -- 172.21.15.194:0/3703874879 >> 172.21.15.194:0/3703874879 conn(0x7f00580f1b90 msgr2=0x7f00580f3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:49:18.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:18.739+0000 7f005caba700 1 -- 172.21.15.194:0/3703874879 shutdown_connections 2024-01-18T06:49:18.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:18.739+0000 7f005caba700 1 -- 172.21.15.194:0/3703874879 wait complete. 2024-01-18T06:49:18.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:18.739+0000 7f005caba700 1 Processor -- start 2024-01-18T06:49:18.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:18.739+0000 7f005caba700 1 -- start start 2024-01-18T06:49:18.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:18.739+0000 7f005caba700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f00580f6430 0x7f005806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:49:18.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:18.739+0000 7f005caba700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0058069e20 con 0x7f00580f6430 2024-01-18T06:49:18.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:18.739+0000 7f005659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f00580f6430 0x7f005806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:49:18.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:18.739+0000 7f005659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f00580f6430 0x7f005806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:53252/0 (socket says 172.21.15.194:53252) 2024-01-18T06:49:18.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:18.739+0000 7f005659c700 1 -- 172.21.15.194:0/2731689682 learned_addr learned my addr 172.21.15.194:0/2731689682 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:49:18.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:18.740+0000 7f005659c700 1 -- 172.21.15.194:0/2731689682 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0040005040 con 0x7f00580f6430 2024-01-18T06:49:18.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:18.740+0000 7f005659c700 1 --2- 172.21.15.194:0/2731689682 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f00580f6430 0x7f005806b9a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f0040013510 tx=0x7f004000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:49:18.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:18.740+0000 7f004effd700 1 -- 172.21.15.194:0/2731689682 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0040009320 con 0x7f00580f6430 2024-01-18T06:49:18.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:18.740+0000 7f005caba700 1 -- 172.21.15.194:0/2731689682 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0058069fc0 con 0x7f00580f6430 2024-01-18T06:49:18.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:18.740+0000 7f004effd700 1 -- 172.21.15.194:0/2731689682 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0040013f00 con 0x7f00580f6430 2024-01-18T06:49:18.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:18.740+0000 7f004effd700 1 -- 172.21.15.194:0/2731689682 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0040014d90 con 0x7f00580f6430 2024-01-18T06:49:18.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:18.740+0000 7f005caba700 1 -- 172.21.15.194:0/2731689682 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f005806a450 con 0x7f00580f6430 2024-01-18T06:49:18.742 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:18.741+0000 7f005caba700 1 -- 172.21.15.194:0/2731689682 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0058045ad0 con 0x7f00580f6430 2024-01-18T06:49:18.742 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:18.741+0000 7f004effd700 1 -- 172.21.15.194:0/2731689682 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f00400095f0 con 0x7f00580f6430 2024-01-18T06:49:18.742 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:18.741+0000 7f004effd700 1 --2- 172.21.15.194:0/2731689682 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f004402ff60 0x7f0044032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:49:18.742 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:18.741+0000 7f004effd700 1 -- 172.21.15.194:0/2731689682 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0040040fe0 con 0x7f00580f6430 2024-01-18T06:49:18.742 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:18.741+0000 7f0055d9b700 1 --2- 172.21.15.194:0/2731689682 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f004402ff60 0x7f0044032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:49:18.742 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:18.742+0000 7f0055d9b700 1 --2- 172.21.15.194:0/2731689682 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f004402ff60 0x7f0044032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f00480096a0 tx=0x7f0048006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:49:18.745 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:18.746+0000 7f004effd700 1 -- 172.21.15.194:0/2731689682 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f004001d7f0 con 0x7f00580f6430 2024-01-18T06:49:18.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:18.953+0000 7f005caba700 1 -- 172.21.15.194:0/2731689682 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f005805ea50 con 0x7f00580f6430 2024-01-18T06:49:18.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:18.954+0000 7f004effd700 1 -- 172.21.15.194:0/2731689682 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0040010230 con 0x7f00580f6430 2024-01-18T06:49:18.954 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:49:18.954 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:49:18.956 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:18.956+0000 7f005caba700 1 -- 172.21.15.194:0/2731689682 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f004402ff60 msgr2=0x7f0044032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:18.956 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:18.956+0000 7f005caba700 1 --2- 172.21.15.194:0/2731689682 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f004402ff60 0x7f0044032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f00480096a0 tx=0x7f0048006b40).stop 2024-01-18T06:49:18.956 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:18.956+0000 7f005caba700 1 -- 172.21.15.194:0/2731689682 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f00580f6430 msgr2=0x7f005806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:18.956 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:18.956+0000 7f005caba700 1 --2- 172.21.15.194:0/2731689682 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f00580f6430 0x7f005806b9a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f0040013510 tx=0x7f004000b040).stop 2024-01-18T06:49:18.956 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:18.956+0000 7f005caba700 1 -- 172.21.15.194:0/2731689682 shutdown_connections 2024-01-18T06:49:18.956 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:18.956+0000 7f005caba700 1 --2- 172.21.15.194:0/2731689682 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f004402ff60 0x7f0044032420 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:18.956 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:18.956+0000 7f005caba700 1 --2- 172.21.15.194:0/2731689682 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f00580f6430 0x7f005806b9a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:18.956 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:18.956+0000 7f005caba700 1 -- 172.21.15.194:0/2731689682 >> 172.21.15.194:0/2731689682 conn(0x7f00580f1b90 msgr2=0x7f00580f2630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:49:18.956 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:18.956+0000 7f005caba700 1 -- 172.21.15.194:0/2731689682 shutdown_connections 2024-01-18T06:49:18.956 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:18.956+0000 7f005caba700 1 -- 172.21.15.194:0/2731689682 wait complete. 2024-01-18T06:49:18.957 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:49:19.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:18 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:49:17.979076+0000 mgr.smithi047.fuhzzr (mgr.14162) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:49:20.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:19 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:49:18.954976+0000 mon.smithi047 (mon.0) 125 : audit [DBG] from='client.? 172.21.15.194:0/2731689682' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:49:20.345 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:49:20.345 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:49:21.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:20 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:49:19.979404+0000 mgr.smithi047.fuhzzr (mgr.14162) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:49:22.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:22.273+0000 7f267dd67700 1 -- 172.21.15.194:0/2008346507 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f26780f47c0 msgr2=0x7f26780f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:22.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:22.273+0000 7f267dd67700 1 --2- 172.21.15.194:0/2008346507 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f26780f47c0 0x7f26780f4be0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f2668009fb0 tx=0x7f2668009b30).stop 2024-01-18T06:49:22.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:22.273+0000 7f267dd67700 1 -- 172.21.15.194:0/2008346507 shutdown_connections 2024-01-18T06:49:22.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:22.273+0000 7f267dd67700 1 --2- 172.21.15.194:0/2008346507 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f26780f47c0 0x7f26780f4be0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:22.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:22.273+0000 7f267dd67700 1 -- 172.21.15.194:0/2008346507 >> 172.21.15.194:0/2008346507 conn(0x7f26780f02e0 msgr2=0x7f26780f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:49:22.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:22.273+0000 7f267dd67700 1 -- 172.21.15.194:0/2008346507 shutdown_connections 2024-01-18T06:49:22.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:22.273+0000 7f267dd67700 1 -- 172.21.15.194:0/2008346507 wait complete. 2024-01-18T06:49:22.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:22.273+0000 7f267dd67700 1 Processor -- start 2024-01-18T06:49:22.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:22.274+0000 7f267dd67700 1 -- start start 2024-01-18T06:49:22.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:22.274+0000 7f267dd67700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f26780f47c0 0x7f2678106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:49:22.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:22.274+0000 7f267dd67700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2678106bf0 con 0x7f26780f47c0 2024-01-18T06:49:22.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:22.274+0000 7f26777fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f26780f47c0 0x7f2678106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:49:22.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:22.274+0000 7f26777fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f26780f47c0 0x7f2678106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:53280/0 (socket says 172.21.15.194:53280) 2024-01-18T06:49:22.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:22.274+0000 7f26777fe700 1 -- 172.21.15.194:0/454766821 learned_addr learned my addr 172.21.15.194:0/454766821 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:49:22.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:22.274+0000 7f26777fe700 1 -- 172.21.15.194:0/454766821 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2668005040 con 0x7f26780f47c0 2024-01-18T06:49:22.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:22.275+0000 7f26777fe700 1 --2- 172.21.15.194:0/454766821 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f26780f47c0 0x7f2678106700 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f2668013510 tx=0x7f266800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:49:22.276 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:22.275+0000 7f265ffff700 1 -- 172.21.15.194:0/454766821 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2668009320 con 0x7f26780f47c0 2024-01-18T06:49:22.276 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:22.275+0000 7f267dd67700 1 -- 172.21.15.194:0/454766821 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2678106df0 con 0x7f26780f47c0 2024-01-18T06:49:22.277 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:22.275+0000 7f265ffff700 1 -- 172.21.15.194:0/454766821 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2668013f00 con 0x7f26780f47c0 2024-01-18T06:49:22.277 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:22.275+0000 7f265ffff700 1 -- 172.21.15.194:0/454766821 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2668014d90 con 0x7f26780f47c0 2024-01-18T06:49:22.277 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:22.275+0000 7f267dd67700 1 -- 172.21.15.194:0/454766821 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2678107280 con 0x7f26780f47c0 2024-01-18T06:49:22.277 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:22.276+0000 7f265ffff700 1 -- 172.21.15.194:0/454766821 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f26680095f0 con 0x7f26780f47c0 2024-01-18T06:49:22.277 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:22.276+0000 7f267dd67700 1 -- 172.21.15.194:0/454766821 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2678045ad0 con 0x7f26780f47c0 2024-01-18T06:49:22.277 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:22.276+0000 7f265ffff700 1 --2- 172.21.15.194:0/454766821 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f266002ff60 0x7f2660032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:49:22.277 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:22.276+0000 7f265ffff700 1 -- 172.21.15.194:0/454766821 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f26680410c0 con 0x7f26780f47c0 2024-01-18T06:49:22.277 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:22.276+0000 7f2676ffd700 1 --2- 172.21.15.194:0/454766821 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f266002ff60 0x7f2660032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:49:22.277 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:22.276+0000 7f2676ffd700 1 --2- 172.21.15.194:0/454766821 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f266002ff60 0x7f2660032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f266c0096a0 tx=0x7f266c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:49:22.280 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:22.280+0000 7f265ffff700 1 -- 172.21.15.194:0/454766821 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f266801d7f0 con 0x7f26780f47c0 2024-01-18T06:49:22.483 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:22.483+0000 7f267dd67700 1 -- 172.21.15.194:0/454766821 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2678108130 con 0x7f26780f47c0 2024-01-18T06:49:22.483 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:22.483+0000 7f265ffff700 1 -- 172.21.15.194:0/454766821 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2668010230 con 0x7f26780f47c0 2024-01-18T06:49:22.484 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:49:22.484 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:49:22.484 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:22.485+0000 7f267dd67700 1 -- 172.21.15.194:0/454766821 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f266002ff60 msgr2=0x7f2660032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:22.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:22.485+0000 7f267dd67700 1 --2- 172.21.15.194:0/454766821 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f266002ff60 0x7f2660032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f266c0096a0 tx=0x7f266c006b40).stop 2024-01-18T06:49:22.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:22.485+0000 7f267dd67700 1 -- 172.21.15.194:0/454766821 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f26780f47c0 msgr2=0x7f2678106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:22.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:22.485+0000 7f267dd67700 1 --2- 172.21.15.194:0/454766821 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f26780f47c0 0x7f2678106700 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f2668013510 tx=0x7f266800b040).stop 2024-01-18T06:49:22.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:22.485+0000 7f267dd67700 1 -- 172.21.15.194:0/454766821 shutdown_connections 2024-01-18T06:49:22.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:22.485+0000 7f267dd67700 1 --2- 172.21.15.194:0/454766821 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f266002ff60 0x7f2660032420 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:22.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:22.485+0000 7f267dd67700 1 --2- 172.21.15.194:0/454766821 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f26780f47c0 0x7f2678106700 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:22.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:22.485+0000 7f267dd67700 1 -- 172.21.15.194:0/454766821 >> 172.21.15.194:0/454766821 conn(0x7f26780f02e0 msgr2=0x7f26780f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:49:22.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:22.485+0000 7f267dd67700 1 -- 172.21.15.194:0/454766821 shutdown_connections 2024-01-18T06:49:22.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:22.485+0000 7f267dd67700 1 -- 172.21.15.194:0/454766821 wait complete. 2024-01-18T06:49:22.485 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:49:23.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:23 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:49:21.979752+0000 mgr.smithi047.fuhzzr (mgr.14162) 27 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:49:23.464 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:23 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:49:22.484194+0000 mon.smithi047 (mon.0) 126 : audit [DBG] from='client.? 172.21.15.194:0/454766821' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:49:23.883 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:49:23.884 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:49:25.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:25.048+0000 7f8c739a0700 1 -- 172.21.15.194:0/3064438473 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c6c0f5330 msgr2=0x7f8c6c0f5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:25.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:25.048+0000 7f8c739a0700 1 --2- 172.21.15.194:0/3064438473 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c6c0f5330 0x7f8c6c0f5750 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f8c68009fb0 tx=0x7f8c68009b30).stop 2024-01-18T06:49:25.050 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:25.048+0000 7f8c739a0700 1 -- 172.21.15.194:0/3064438473 shutdown_connections 2024-01-18T06:49:25.050 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:25.048+0000 7f8c739a0700 1 --2- 172.21.15.194:0/3064438473 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c6c0f5330 0x7f8c6c0f5750 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:25.050 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:25.048+0000 7f8c739a0700 1 -- 172.21.15.194:0/3064438473 >> 172.21.15.194:0/3064438473 conn(0x7f8c6c0f0a90 msgr2=0x7f8c6c0f2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:49:25.050 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:25.048+0000 7f8c739a0700 1 -- 172.21.15.194:0/3064438473 shutdown_connections 2024-01-18T06:49:25.050 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:25.048+0000 7f8c739a0700 1 -- 172.21.15.194:0/3064438473 wait complete. 2024-01-18T06:49:25.050 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:25.048+0000 7f8c739a0700 1 Processor -- start 2024-01-18T06:49:25.050 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:25.049+0000 7f8c739a0700 1 -- start start 2024-01-18T06:49:25.050 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:25.049+0000 7f8c739a0700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c6c0f5330 0x7f8c6c10baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:49:25.051 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:25.049+0000 7f8c739a0700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6c1074e0 con 0x7f8c6c0f5330 2024-01-18T06:49:25.051 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:25.049+0000 7f8c7173c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c6c0f5330 0x7f8c6c10baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:49:25.051 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:25.049+0000 7f8c7173c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c6c0f5330 0x7f8c6c10baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:53288/0 (socket says 172.21.15.194:53288) 2024-01-18T06:49:25.051 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:25.049+0000 7f8c7173c700 1 -- 172.21.15.194:0/342622694 learned_addr learned my addr 172.21.15.194:0/342622694 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:49:25.051 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:25.049+0000 7f8c7173c700 1 -- 172.21.15.194:0/342622694 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c68005040 con 0x7f8c6c0f5330 2024-01-18T06:49:25.051 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:25.049+0000 7f8c7173c700 1 --2- 172.21.15.194:0/342622694 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c6c0f5330 0x7f8c6c10baf0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f8c68013510 tx=0x7f8c6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:49:25.051 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:25.050+0000 7f8c61ffb700 1 -- 172.21.15.194:0/342622694 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c68009320 con 0x7f8c6c0f5330 2024-01-18T06:49:25.051 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:25.050+0000 7f8c61ffb700 1 -- 172.21.15.194:0/342622694 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8c68013f00 con 0x7f8c6c0f5330 2024-01-18T06:49:25.051 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:25.050+0000 7f8c739a0700 1 -- 172.21.15.194:0/342622694 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c6c060220 con 0x7f8c6c0f5330 2024-01-18T06:49:25.051 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:25.050+0000 7f8c61ffb700 1 -- 172.21.15.194:0/342622694 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c68014d90 con 0x7f8c6c0f5330 2024-01-18T06:49:25.051 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:25.050+0000 7f8c739a0700 1 -- 172.21.15.194:0/342622694 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c6c10bfe0 con 0x7f8c6c0f5330 2024-01-18T06:49:25.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:25.051+0000 7f8c739a0700 1 -- 172.21.15.194:0/342622694 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c6c045ad0 con 0x7f8c6c0f5330 2024-01-18T06:49:25.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:25.051+0000 7f8c61ffb700 1 -- 172.21.15.194:0/342622694 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8c680095f0 con 0x7f8c6c0f5330 2024-01-18T06:49:25.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:25.051+0000 7f8c61ffb700 1 --2- 172.21.15.194:0/342622694 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f8c5802ff60 0x7f8c58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:49:25.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:25.051+0000 7f8c61ffb700 1 -- 172.21.15.194:0/342622694 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8c680410c0 con 0x7f8c6c0f5330 2024-01-18T06:49:25.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:25.051+0000 7f8c70f3b700 1 --2- 172.21.15.194:0/342622694 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f8c5802ff60 0x7f8c58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:49:25.053 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:25.051+0000 7f8c70f3b700 1 --2- 172.21.15.194:0/342622694 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f8c5802ff60 0x7f8c58032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f8c5c0096a0 tx=0x7f8c5c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:49:25.055 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:25.056+0000 7f8c61ffb700 1 -- 172.21.15.194:0/342622694 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8c6801d7f0 con 0x7f8c6c0f5330 2024-01-18T06:49:25.261 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:25.261+0000 7f8c739a0700 1 -- 172.21.15.194:0/342622694 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8c6c05ea50 con 0x7f8c6c0f5330 2024-01-18T06:49:25.262 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:25.262+0000 7f8c61ffb700 1 -- 172.21.15.194:0/342622694 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8c68010230 con 0x7f8c6c0f5330 2024-01-18T06:49:25.262 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:49:25.262 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:49:25.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:25.263+0000 7f8c739a0700 1 -- 172.21.15.194:0/342622694 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f8c5802ff60 msgr2=0x7f8c58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:25.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:25.263+0000 7f8c739a0700 1 --2- 172.21.15.194:0/342622694 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f8c5802ff60 0x7f8c58032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f8c5c0096a0 tx=0x7f8c5c006b40).stop 2024-01-18T06:49:25.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:25.263+0000 7f8c739a0700 1 -- 172.21.15.194:0/342622694 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c6c0f5330 msgr2=0x7f8c6c10baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:25.264 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:25.263+0000 7f8c739a0700 1 --2- 172.21.15.194:0/342622694 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c6c0f5330 0x7f8c6c10baf0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f8c68013510 tx=0x7f8c6800b040).stop 2024-01-18T06:49:25.264 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:25.264+0000 7f8c739a0700 1 -- 172.21.15.194:0/342622694 shutdown_connections 2024-01-18T06:49:25.264 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:25.264+0000 7f8c739a0700 1 --2- 172.21.15.194:0/342622694 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f8c5802ff60 0x7f8c58032420 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:25.264 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:25.264+0000 7f8c739a0700 1 --2- 172.21.15.194:0/342622694 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c6c0f5330 0x7f8c6c10baf0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:25.264 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:25.264+0000 7f8c739a0700 1 -- 172.21.15.194:0/342622694 >> 172.21.15.194:0/342622694 conn(0x7f8c6c0f0a90 msgr2=0x7f8c6c0f1530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:49:25.264 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:25.264+0000 7f8c739a0700 1 -- 172.21.15.194:0/342622694 shutdown_connections 2024-01-18T06:49:25.264 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:25.264+0000 7f8c739a0700 1 -- 172.21.15.194:0/342622694 wait complete. 2024-01-18T06:49:25.265 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:49:25.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:25 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:49:23.980117+0000 mgr.smithi047.fuhzzr (mgr.14162) 28 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:49:26.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:26 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:49:25.262994+0000 mon.smithi047 (mon.0) 127 : audit [DBG] from='client.? 172.21.15.194:0/342622694' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:49:26.673 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:49:26.673 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:49:27.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:27 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:49:25.980553+0000 mgr.smithi047.fuhzzr (mgr.14162) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:49:29.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:28 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:49:27.980938+0000 mgr.smithi047.fuhzzr (mgr.14162) 30 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:49:31.464 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:31 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:49:29.981365+0000 mgr.smithi047.fuhzzr (mgr.14162) 31 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:49:31.464 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:31 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:49:30.111296+0000 mon.smithi047 (mon.0) 128 : audit [INF] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.fuhzzr/mirror_snapshot_schedule"}]: dispatch 2024-01-18T06:49:31.464 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:31 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:49:30.113723+0000 mon.smithi047 (mon.0) 129 : audit [INF] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.fuhzzr/trash_purge_schedule"}]: dispatch 2024-01-18T06:49:31.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:31.507+0000 7ff23a031700 1 -- 172.21.15.194:0/1692233416 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2340f5410 msgr2=0x7ff2340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:31.614 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:31.507+0000 7ff23a031700 1 --2- 172.21.15.194:0/1692233416 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2340f5410 0x7ff2340f5830 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7ff228009fb0 tx=0x7ff228009b30).stop 2024-01-18T06:49:31.614 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:31.507+0000 7ff23a031700 1 -- 172.21.15.194:0/1692233416 shutdown_connections 2024-01-18T06:49:31.614 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:31.507+0000 7ff23a031700 1 --2- 172.21.15.194:0/1692233416 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2340f5410 0x7ff2340f5830 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:31.614 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:31.507+0000 7ff23a031700 1 -- 172.21.15.194:0/1692233416 >> 172.21.15.194:0/1692233416 conn(0x7ff2340f0b70 msgr2=0x7ff2340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:49:31.614 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:31.507+0000 7ff23a031700 1 -- 172.21.15.194:0/1692233416 shutdown_connections 2024-01-18T06:49:31.615 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:31.507+0000 7ff23a031700 1 -- 172.21.15.194:0/1692233416 wait complete. 2024-01-18T06:49:31.615 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:31.508+0000 7ff23a031700 1 Processor -- start 2024-01-18T06:49:31.615 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:31.508+0000 7ff23a031700 1 -- start start 2024-01-18T06:49:31.615 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:31.508+0000 7ff23a031700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2340f5410 0x7ff234106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:49:31.615 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:31.508+0000 7ff23a031700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff234106c00 con 0x7ff2340f5410 2024-01-18T06:49:31.615 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:31.509+0000 7ff2337fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2340f5410 0x7ff234106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:49:31.615 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:31.509+0000 7ff2337fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2340f5410 0x7ff234106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:39300/0 (socket says 172.21.15.194:39300) 2024-01-18T06:49:31.616 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:31.509+0000 7ff2337fe700 1 -- 172.21.15.194:0/4230174439 learned_addr learned my addr 172.21.15.194:0/4230174439 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:49:31.616 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:31.509+0000 7ff2337fe700 1 -- 172.21.15.194:0/4230174439 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff228005040 con 0x7ff2340f5410 2024-01-18T06:49:31.616 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:31.509+0000 7ff2337fe700 1 --2- 172.21.15.194:0/4230174439 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2340f5410 0x7ff234106710 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7ff228013510 tx=0x7ff22800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:49:31.616 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:31.510+0000 7ff21bfff700 1 -- 172.21.15.194:0/4230174439 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff228009320 con 0x7ff2340f5410 2024-01-18T06:49:31.616 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:31.510+0000 7ff23a031700 1 -- 172.21.15.194:0/4230174439 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff234106e00 con 0x7ff2340f5410 2024-01-18T06:49:31.616 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:31.510+0000 7ff23a031700 1 -- 172.21.15.194:0/4230174439 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff234107290 con 0x7ff2340f5410 2024-01-18T06:49:31.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:31.510+0000 7ff21bfff700 1 -- 172.21.15.194:0/4230174439 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff228013f00 con 0x7ff2340f5410 2024-01-18T06:49:31.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:31.510+0000 7ff21bfff700 1 -- 172.21.15.194:0/4230174439 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff228014d60 con 0x7ff2340f5410 2024-01-18T06:49:31.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:31.511+0000 7ff21bfff700 1 -- 172.21.15.194:0/4230174439 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff2280263a0 con 0x7ff2340f5410 2024-01-18T06:49:31.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:31.511+0000 7ff23a031700 1 -- 172.21.15.194:0/4230174439 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff234045ad0 con 0x7ff2340f5410 2024-01-18T06:49:31.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:31.511+0000 7ff21bfff700 1 --2- 172.21.15.194:0/4230174439 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7ff21c02ff60 0x7ff21c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:49:31.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:31.511+0000 7ff21bfff700 1 -- 172.21.15.194:0/4230174439 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff228041490 con 0x7ff2340f5410 2024-01-18T06:49:31.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:31.511+0000 7ff232ffd700 1 --2- 172.21.15.194:0/4230174439 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7ff21c02ff60 0x7ff21c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:49:31.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:31.512+0000 7ff232ffd700 1 --2- 172.21.15.194:0/4230174439 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7ff21c02ff60 0x7ff21c032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7ff2240096a0 tx=0x7ff224006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:49:31.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:31.515+0000 7ff21bfff700 1 -- 172.21.15.194:0/4230174439 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff228012510 con 0x7ff2340f5410 2024-01-18T06:49:31.725 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:31.725+0000 7ff23a031700 1 -- 172.21.15.194:0/4230174439 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff2340fae60 con 0x7ff2340f5410 2024-01-18T06:49:31.726 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:31.726+0000 7ff21bfff700 1 -- 172.21.15.194:0/4230174439 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff228012330 con 0x7ff2340f5410 2024-01-18T06:49:31.726 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:49:31.726 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:49:31.727 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:31.727+0000 7ff23a031700 1 -- 172.21.15.194:0/4230174439 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7ff21c02ff60 msgr2=0x7ff21c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:31.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:31.728+0000 7ff23a031700 1 --2- 172.21.15.194:0/4230174439 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7ff21c02ff60 0x7ff21c032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7ff2240096a0 tx=0x7ff224006b40).stop 2024-01-18T06:49:31.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:31.728+0000 7ff23a031700 1 -- 172.21.15.194:0/4230174439 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2340f5410 msgr2=0x7ff234106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:31.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:31.728+0000 7ff23a031700 1 --2- 172.21.15.194:0/4230174439 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2340f5410 0x7ff234106710 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7ff228013510 tx=0x7ff22800b040).stop 2024-01-18T06:49:31.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:31.728+0000 7ff23a031700 1 -- 172.21.15.194:0/4230174439 shutdown_connections 2024-01-18T06:49:31.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:31.728+0000 7ff23a031700 1 --2- 172.21.15.194:0/4230174439 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7ff21c02ff60 0x7ff21c032420 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:31.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:31.728+0000 7ff23a031700 1 --2- 172.21.15.194:0/4230174439 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2340f5410 0x7ff234106710 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:31.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:31.728+0000 7ff23a031700 1 -- 172.21.15.194:0/4230174439 >> 172.21.15.194:0/4230174439 conn(0x7ff2340f0b70 msgr2=0x7ff23405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:49:31.729 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:31.728+0000 7ff23a031700 1 -- 172.21.15.194:0/4230174439 shutdown_connections 2024-01-18T06:49:31.729 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:31.728+0000 7ff23a031700 1 -- 172.21.15.194:0/4230174439 wait complete. 2024-01-18T06:49:31.729 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:49:32.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:32 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:49:31.726834+0000 mon.smithi047 (mon.0) 130 : audit [DBG] from='client.? 172.21.15.194:0/4230174439' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:49:33.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:33 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:49:31.981771+0000 mgr.smithi047.fuhzzr (mgr.14162) 32 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:49:34.642 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:49:34.642 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:49:35.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:35 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:49:33.982044+0000 mgr.smithi047.fuhzzr (mgr.14162) 33 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:49:35.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:35.883+0000 7f84f7866700 1 -- 172.21.15.194:0/1979074889 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f84f00f4b80 msgr2=0x7f84f00f4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:35.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:35.883+0000 7f84f7866700 1 --2- 172.21.15.194:0/1979074889 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f84f00f4b80 0x7f84f00f4fa0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f84ec009fb0 tx=0x7f84ec009b30).stop 2024-01-18T06:49:35.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:35.883+0000 7f84f7866700 1 -- 172.21.15.194:0/1979074889 shutdown_connections 2024-01-18T06:49:35.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:35.883+0000 7f84f7866700 1 --2- 172.21.15.194:0/1979074889 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f84f00f4b80 0x7f84f00f4fa0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:35.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:35.883+0000 7f84f7866700 1 -- 172.21.15.194:0/1979074889 >> 172.21.15.194:0/1979074889 conn(0x7f84f00f02e0 msgr2=0x7f84f00f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:49:35.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:35.883+0000 7f84f7866700 1 -- 172.21.15.194:0/1979074889 shutdown_connections 2024-01-18T06:49:35.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:35.883+0000 7f84f7866700 1 -- 172.21.15.194:0/1979074889 wait complete. 2024-01-18T06:49:35.885 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:35.884+0000 7f84f7866700 1 Processor -- start 2024-01-18T06:49:35.885 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:35.884+0000 7f84f7866700 1 -- start start 2024-01-18T06:49:35.885 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:35.884+0000 7f84f7866700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f84f00f4b80 0x7f84f0104e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:49:35.885 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:35.884+0000 7f84f7866700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f0105300 con 0x7f84f00f4b80 2024-01-18T06:49:35.885 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:35.884+0000 7f84f5602700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f84f00f4b80 0x7f84f0104e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:49:35.885 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:35.884+0000 7f84f5602700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f84f00f4b80 0x7f84f0104e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:39318/0 (socket says 172.21.15.194:39318) 2024-01-18T06:49:35.885 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:35.884+0000 7f84f5602700 1 -- 172.21.15.194:0/3253340634 learned_addr learned my addr 172.21.15.194:0/3253340634 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:49:35.885 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:35.884+0000 7f84f5602700 1 -- 172.21.15.194:0/3253340634 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84ec005040 con 0x7f84f00f4b80 2024-01-18T06:49:35.885 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:35.884+0000 7f84f5602700 1 --2- 172.21.15.194:0/3253340634 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f84f00f4b80 0x7f84f0104e10 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f84ec013510 tx=0x7f84ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:49:35.885 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:35.885+0000 7f84e5ffb700 1 -- 172.21.15.194:0/3253340634 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84ec009320 con 0x7f84f00f4b80 2024-01-18T06:49:35.885 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:35.885+0000 7f84e5ffb700 1 -- 172.21.15.194:0/3253340634 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f84ec013f00 con 0x7f84f00f4b80 2024-01-18T06:49:35.885 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:35.885+0000 7f84f7866700 1 -- 172.21.15.194:0/3253340634 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84f0105500 con 0x7f84f00f4b80 2024-01-18T06:49:35.886 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:35.885+0000 7f84e5ffb700 1 -- 172.21.15.194:0/3253340634 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84ec014d90 con 0x7f84f00f4b80 2024-01-18T06:49:35.886 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:35.885+0000 7f84f7866700 1 -- 172.21.15.194:0/3253340634 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84f005ed00 con 0x7f84f00f4b80 2024-01-18T06:49:35.886 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:35.886+0000 7f84e5ffb700 1 -- 172.21.15.194:0/3253340634 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f84ec0095f0 con 0x7f84f00f4b80 2024-01-18T06:49:35.886 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:35.886+0000 7f84f7866700 1 -- 172.21.15.194:0/3253340634 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84f0105990 con 0x7f84f00f4b80 2024-01-18T06:49:35.887 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:35.886+0000 7f84e5ffb700 1 --2- 172.21.15.194:0/3253340634 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f84dc02ff60 0x7f84dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:49:35.887 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:35.886+0000 7f84e5ffb700 1 -- 172.21.15.194:0/3253340634 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f84ec041f00 con 0x7f84f00f4b80 2024-01-18T06:49:35.887 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:35.886+0000 7f84f4e01700 1 --2- 172.21.15.194:0/3253340634 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f84dc02ff60 0x7f84dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:49:35.887 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:35.887+0000 7f84f4e01700 1 --2- 172.21.15.194:0/3253340634 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f84dc02ff60 0x7f84dc032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f84e00096a0 tx=0x7f84e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:49:35.890 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:35.891+0000 7f84e5ffb700 1 -- 172.21.15.194:0/3253340634 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f84f0105990 con 0x7f84f00f4b80 2024-01-18T06:49:36.092 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:36.093+0000 7f84f7866700 1 -- 172.21.15.194:0/3253340634 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f84f0105990 con 0x7f84f00f4b80 2024-01-18T06:49:36.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:36.094+0000 7f84e5ffb700 1 -- 172.21.15.194:0/3253340634 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f84f0105990 con 0x7f84f00f4b80 2024-01-18T06:49:36.093 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:49:36.093 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:49:36.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:36.095+0000 7f84f7866700 1 -- 172.21.15.194:0/3253340634 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f84dc02ff60 msgr2=0x7f84dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:36.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:36.095+0000 7f84f7866700 1 --2- 172.21.15.194:0/3253340634 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f84dc02ff60 0x7f84dc032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f84e00096a0 tx=0x7f84e0006b40).stop 2024-01-18T06:49:36.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:36.095+0000 7f84f7866700 1 -- 172.21.15.194:0/3253340634 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f84f00f4b80 msgr2=0x7f84f0104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:36.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:36.095+0000 7f84f7866700 1 --2- 172.21.15.194:0/3253340634 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f84f00f4b80 0x7f84f0104e10 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f84ec013510 tx=0x7f84ec00b040).stop 2024-01-18T06:49:36.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:36.095+0000 7f84f7866700 1 -- 172.21.15.194:0/3253340634 shutdown_connections 2024-01-18T06:49:36.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:36.095+0000 7f84f7866700 1 --2- 172.21.15.194:0/3253340634 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f84dc02ff60 0x7f84dc032420 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:36.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:36.095+0000 7f84f7866700 1 --2- 172.21.15.194:0/3253340634 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f84f00f4b80 0x7f84f0104e10 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:36.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:36.095+0000 7f84f7866700 1 -- 172.21.15.194:0/3253340634 >> 172.21.15.194:0/3253340634 conn(0x7f84f00f02e0 msgr2=0x7f84f00f0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:49:36.096 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:36.095+0000 7f84f7866700 1 -- 172.21.15.194:0/3253340634 shutdown_connections 2024-01-18T06:49:36.096 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:36.095+0000 7f84f7866700 1 -- 172.21.15.194:0/3253340634 wait complete. 2024-01-18T06:49:36.096 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:49:37.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:49:35.982460+0000 mgr.smithi047.fuhzzr (mgr.14162) 34 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:49:37.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:49:36.094543+0000 mon.smithi047 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.194:0/3253340634' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:49:37.490 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:49:37.490 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:49:38.685 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:38.685+0000 7fda2d922700 1 -- 172.21.15.194:0/676970677 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda28060280 msgr2=0x7fda280f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:38.685 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:38.685+0000 7fda2d922700 1 --2- 172.21.15.194:0/676970677 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda28060280 0x7fda280f90f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fda1c009fb0 tx=0x7fda1c009b30).stop 2024-01-18T06:49:38.685 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:38.685+0000 7fda2d922700 1 -- 172.21.15.194:0/676970677 shutdown_connections 2024-01-18T06:49:38.685 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:38.685+0000 7fda2d922700 1 --2- 172.21.15.194:0/676970677 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda28060280 0x7fda280f90f0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:38.685 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:38.685+0000 7fda2d922700 1 -- 172.21.15.194:0/676970677 >> 172.21.15.194:0/676970677 conn(0x7fda280f0b70 msgr2=0x7fda280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:49:38.685 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:38.685+0000 7fda2d922700 1 -- 172.21.15.194:0/676970677 shutdown_connections 2024-01-18T06:49:38.685 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:38.685+0000 7fda2d922700 1 -- 172.21.15.194:0/676970677 wait complete. 2024-01-18T06:49:38.687 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:38.686+0000 7fda2d922700 1 Processor -- start 2024-01-18T06:49:38.687 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:38.686+0000 7fda2d922700 1 -- start start 2024-01-18T06:49:38.687 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:38.686+0000 7fda2d922700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda28060280 0x7fda281066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:49:38.687 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:38.686+0000 7fda2d922700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda28106bd0 con 0x7fda28060280 2024-01-18T06:49:38.687 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:38.686+0000 7fda26ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda28060280 0x7fda281066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:49:38.687 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:38.686+0000 7fda26ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda28060280 0x7fda281066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:49694/0 (socket says 172.21.15.194:49694) 2024-01-18T06:49:38.687 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:38.686+0000 7fda26ffd700 1 -- 172.21.15.194:0/2051107919 learned_addr learned my addr 172.21.15.194:0/2051107919 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:49:38.687 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:38.686+0000 7fda26ffd700 1 -- 172.21.15.194:0/2051107919 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda1c005040 con 0x7fda28060280 2024-01-18T06:49:38.687 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:38.687+0000 7fda26ffd700 1 --2- 172.21.15.194:0/2051107919 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda28060280 0x7fda281066e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fda1c013510 tx=0x7fda1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:49:38.687 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:38.687+0000 7fda0f7fe700 1 -- 172.21.15.194:0/2051107919 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda1c009320 con 0x7fda28060280 2024-01-18T06:49:38.687 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:38.687+0000 7fda2d922700 1 -- 172.21.15.194:0/2051107919 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda28106dd0 con 0x7fda28060280 2024-01-18T06:49:38.687 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:38.687+0000 7fda0f7fe700 1 -- 172.21.15.194:0/2051107919 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fda1c013f00 con 0x7fda28060280 2024-01-18T06:49:38.687 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:38.687+0000 7fda0f7fe700 1 -- 172.21.15.194:0/2051107919 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda1c014dc0 con 0x7fda28060280 2024-01-18T06:49:38.688 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:38.687+0000 7fda2d922700 1 -- 172.21.15.194:0/2051107919 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda28107260 con 0x7fda28060280 2024-01-18T06:49:38.694 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:38.688+0000 7fda0f7fe700 1 -- 172.21.15.194:0/2051107919 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fda1c0095f0 con 0x7fda28060280 2024-01-18T06:49:38.694 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:38.688+0000 7fda2d922700 1 -- 172.21.15.194:0/2051107919 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda28045ad0 con 0x7fda28060280 2024-01-18T06:49:38.694 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:38.688+0000 7fda0f7fe700 1 --2- 172.21.15.194:0/2051107919 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fda1002ffb0 0x7fda10032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:49:38.694 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:38.688+0000 7fda0f7fe700 1 -- 172.21.15.194:0/2051107919 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fda1c0410a0 con 0x7fda28060280 2024-01-18T06:49:38.694 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:38.688+0000 7fda267fc700 1 --2- 172.21.15.194:0/2051107919 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fda1002ffb0 0x7fda10032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:49:38.695 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:38.688+0000 7fda267fc700 1 --2- 172.21.15.194:0/2051107919 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fda1002ffb0 0x7fda10032470 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fda180096a0 tx=0x7fda18006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:49:38.695 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:38.693+0000 7fda0f7fe700 1 -- 172.21.15.194:0/2051107919 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fda1c06d2b0 con 0x7fda28060280 2024-01-18T06:49:38.896 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:38.897+0000 7fda2d922700 1 -- 172.21.15.194:0/2051107919 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fda28108110 con 0x7fda28060280 2024-01-18T06:49:38.897 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:38.898+0000 7fda0f7fe700 1 -- 172.21.15.194:0/2051107919 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fda1c014420 con 0x7fda28060280 2024-01-18T06:49:38.897 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:49:38.897 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:49:38.899 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:38.899+0000 7fda2d922700 1 -- 172.21.15.194:0/2051107919 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fda1002ffb0 msgr2=0x7fda10032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:38.899 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:38.899+0000 7fda2d922700 1 --2- 172.21.15.194:0/2051107919 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fda1002ffb0 0x7fda10032470 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fda180096a0 tx=0x7fda18006b40).stop 2024-01-18T06:49:38.899 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:38.899+0000 7fda2d922700 1 -- 172.21.15.194:0/2051107919 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda28060280 msgr2=0x7fda281066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:38.899 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:38.899+0000 7fda2d922700 1 --2- 172.21.15.194:0/2051107919 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda28060280 0x7fda281066e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fda1c013510 tx=0x7fda1c00b040).stop 2024-01-18T06:49:38.899 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:38.899+0000 7fda2d922700 1 -- 172.21.15.194:0/2051107919 shutdown_connections 2024-01-18T06:49:38.899 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:38.899+0000 7fda2d922700 1 --2- 172.21.15.194:0/2051107919 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fda1002ffb0 0x7fda10032470 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:38.899 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:38.899+0000 7fda2d922700 1 --2- 172.21.15.194:0/2051107919 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda28060280 0x7fda281066e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:38.899 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:38.899+0000 7fda2d922700 1 -- 172.21.15.194:0/2051107919 >> 172.21.15.194:0/2051107919 conn(0x7fda280f0b70 msgr2=0x7fda280f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:49:38.899 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:38.899+0000 7fda2d922700 1 -- 172.21.15.194:0/2051107919 shutdown_connections 2024-01-18T06:49:38.899 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:38.899+0000 7fda2d922700 1 -- 172.21.15.194:0/2051107919 wait complete. 2024-01-18T06:49:38.900 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:49:39.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:38 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:49:37.982857+0000 mgr.smithi047.fuhzzr (mgr.14162) 35 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:49:39.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:38 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:49:38.898525+0000 mon.smithi047 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.194:0/2051107919' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:49:40.290 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:49:40.290 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:49:41.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:41 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:49:39.983102+0000 mgr.smithi047.fuhzzr (mgr.14162) 36 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:49:41.479 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:41.478+0000 7f3c94f60700 1 -- 172.21.15.194:0/412252934 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3c900f5410 msgr2=0x7f3c900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:41.479 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:41.478+0000 7f3c94f60700 1 --2- 172.21.15.194:0/412252934 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3c900f5410 0x7f3c900f5830 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f3c84009fb0 tx=0x7f3c84009b30).stop 2024-01-18T06:49:41.479 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:41.478+0000 7f3c94f60700 1 -- 172.21.15.194:0/412252934 shutdown_connections 2024-01-18T06:49:41.479 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:41.478+0000 7f3c94f60700 1 --2- 172.21.15.194:0/412252934 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3c900f5410 0x7f3c900f5830 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:41.480 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:41.478+0000 7f3c94f60700 1 -- 172.21.15.194:0/412252934 >> 172.21.15.194:0/412252934 conn(0x7f3c900f0b70 msgr2=0x7f3c900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:49:41.480 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:41.478+0000 7f3c94f60700 1 -- 172.21.15.194:0/412252934 shutdown_connections 2024-01-18T06:49:41.480 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:41.478+0000 7f3c94f60700 1 -- 172.21.15.194:0/412252934 wait complete. 2024-01-18T06:49:41.480 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:41.479+0000 7f3c94f60700 1 Processor -- start 2024-01-18T06:49:41.480 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:41.479+0000 7f3c94f60700 1 -- start start 2024-01-18T06:49:41.480 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:41.479+0000 7f3c94f60700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3c900f5410 0x7f3c901066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:49:41.480 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:41.479+0000 7f3c94f60700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c90106bd0 con 0x7f3c900f5410 2024-01-18T06:49:41.480 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:41.479+0000 7f3c8e59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3c900f5410 0x7f3c901066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:49:41.480 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:41.479+0000 7f3c8e59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3c900f5410 0x7f3c901066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:49702/0 (socket says 172.21.15.194:49702) 2024-01-18T06:49:41.480 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:41.479+0000 7f3c8e59c700 1 -- 172.21.15.194:0/2799244959 learned_addr learned my addr 172.21.15.194:0/2799244959 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:49:41.480 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:41.479+0000 7f3c8e59c700 1 -- 172.21.15.194:0/2799244959 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c84005040 con 0x7f3c900f5410 2024-01-18T06:49:41.480 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:41.479+0000 7f3c8e59c700 1 --2- 172.21.15.194:0/2799244959 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3c900f5410 0x7f3c901066e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f3c84014000 tx=0x7f3c8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:49:41.480 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:41.480+0000 7f3c7effd700 1 -- 172.21.15.194:0/2799244959 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c840095f0 con 0x7f3c900f5410 2024-01-18T06:49:41.481 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:41.480+0000 7f3c94f60700 1 -- 172.21.15.194:0/2799244959 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c90106dd0 con 0x7f3c900f5410 2024-01-18T06:49:41.481 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:41.480+0000 7f3c7effd700 1 -- 172.21.15.194:0/2799244959 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3c84007e70 con 0x7f3c900f5410 2024-01-18T06:49:41.481 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:41.480+0000 7f3c7effd700 1 -- 172.21.15.194:0/2799244959 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c84014ea0 con 0x7f3c900f5410 2024-01-18T06:49:41.481 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:41.480+0000 7f3c94f60700 1 -- 172.21.15.194:0/2799244959 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c90107260 con 0x7f3c900f5410 2024-01-18T06:49:41.481 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:41.480+0000 7f3c7effd700 1 -- 172.21.15.194:0/2799244959 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3c84025070 con 0x7f3c900f5410 2024-01-18T06:49:41.482 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:41.480+0000 7f3c94f60700 1 -- 172.21.15.194:0/2799244959 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c90045ad0 con 0x7f3c900f5410 2024-01-18T06:49:41.482 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:41.481+0000 7f3c7effd700 1 --2- 172.21.15.194:0/2799244959 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3c7802ff60 0x7f3c78032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:49:41.482 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:41.481+0000 7f3c7effd700 1 -- 172.21.15.194:0/2799244959 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3c840409c0 con 0x7f3c900f5410 2024-01-18T06:49:41.482 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:41.481+0000 7f3c8dd9b700 1 --2- 172.21.15.194:0/2799244959 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3c7802ff60 0x7f3c78032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:49:41.482 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:41.481+0000 7f3c8dd9b700 1 --2- 172.21.15.194:0/2799244959 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3c7802ff60 0x7f3c78032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f3c800096a0 tx=0x7f3c80006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:49:41.484 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:41.485+0000 7f3c7effd700 1 -- 172.21.15.194:0/2799244959 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3c840079c0 con 0x7f3c900f5410 2024-01-18T06:49:41.689 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:41.690+0000 7f3c94f60700 1 -- 172.21.15.194:0/2799244959 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3c901080b0 con 0x7f3c900f5410 2024-01-18T06:49:41.690 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:41.690+0000 7f3c7effd700 1 -- 172.21.15.194:0/2799244959 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3c8400f300 con 0x7f3c900f5410 2024-01-18T06:49:41.690 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:49:41.690 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:49:41.691 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:41.692+0000 7f3c94f60700 1 -- 172.21.15.194:0/2799244959 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3c7802ff60 msgr2=0x7f3c78032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:41.691 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:41.692+0000 7f3c94f60700 1 --2- 172.21.15.194:0/2799244959 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3c7802ff60 0x7f3c78032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f3c800096a0 tx=0x7f3c80006b40).stop 2024-01-18T06:49:41.691 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:41.692+0000 7f3c94f60700 1 -- 172.21.15.194:0/2799244959 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3c900f5410 msgr2=0x7f3c901066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:41.692 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:41.692+0000 7f3c94f60700 1 --2- 172.21.15.194:0/2799244959 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3c900f5410 0x7f3c901066e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f3c84014000 tx=0x7f3c8400b040).stop 2024-01-18T06:49:41.692 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:41.692+0000 7f3c94f60700 1 -- 172.21.15.194:0/2799244959 shutdown_connections 2024-01-18T06:49:41.692 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:41.692+0000 7f3c94f60700 1 --2- 172.21.15.194:0/2799244959 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3c7802ff60 0x7f3c78032420 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:41.692 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:41.692+0000 7f3c94f60700 1 --2- 172.21.15.194:0/2799244959 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3c900f5410 0x7f3c901066e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:41.692 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:41.692+0000 7f3c94f60700 1 -- 172.21.15.194:0/2799244959 >> 172.21.15.194:0/2799244959 conn(0x7f3c900f0b70 msgr2=0x7f3c9005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:49:41.692 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:41.692+0000 7f3c94f60700 1 -- 172.21.15.194:0/2799244959 shutdown_connections 2024-01-18T06:49:41.692 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:41.692+0000 7f3c94f60700 1 -- 172.21.15.194:0/2799244959 wait complete. 2024-01-18T06:49:41.692 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:49:42.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:42 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:49:41.691176+0000 mon.smithi047 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.194:0/2799244959' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:49:43.099 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:49:43.100 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:49:43.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:43 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:49:41.983343+0000 mgr.smithi047.fuhzzr (mgr.14162) 37 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:49:44.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:44.296+0000 7f8c1dbfe700 1 -- 172.21.15.194:0/288310171 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c180f62c0 msgr2=0x7f8c180f66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:44.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:44.296+0000 7f8c1dbfe700 1 --2- 172.21.15.194:0/288310171 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c180f62c0 0x7f8c180f66e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f8c0c009fb0 tx=0x7f8c0c009b30).stop 2024-01-18T06:49:44.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:44.296+0000 7f8c1dbfe700 1 -- 172.21.15.194:0/288310171 shutdown_connections 2024-01-18T06:49:44.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:44.296+0000 7f8c1dbfe700 1 --2- 172.21.15.194:0/288310171 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c180f62c0 0x7f8c180f66e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:44.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:44.296+0000 7f8c1dbfe700 1 -- 172.21.15.194:0/288310171 >> 172.21.15.194:0/288310171 conn(0x7f8c180f1ac0 msgr2=0x7f8c180f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:49:44.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:44.296+0000 7f8c1dbfe700 1 -- 172.21.15.194:0/288310171 shutdown_connections 2024-01-18T06:49:44.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:44.296+0000 7f8c1dbfe700 1 -- 172.21.15.194:0/288310171 wait complete. 2024-01-18T06:49:44.298 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:44.297+0000 7f8c1dbfe700 1 Processor -- start 2024-01-18T06:49:44.298 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:44.297+0000 7f8c1dbfe700 1 -- start start 2024-01-18T06:49:44.298 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:44.297+0000 7f8c1dbfe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c180f62c0 0x7f8c18106520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:49:44.298 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:44.297+0000 7f8c1dbfe700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c18106a10 con 0x7f8c180f62c0 2024-01-18T06:49:44.298 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:44.297+0000 7f8c1cbfc700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c180f62c0 0x7f8c18106520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:49:44.298 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:44.297+0000 7f8c1cbfc700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c180f62c0 0x7f8c18106520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:49718/0 (socket says 172.21.15.194:49718) 2024-01-18T06:49:44.299 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:44.297+0000 7f8c1cbfc700 1 -- 172.21.15.194:0/3973085752 learned_addr learned my addr 172.21.15.194:0/3973085752 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:49:44.299 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:44.298+0000 7f8c1cbfc700 1 -- 172.21.15.194:0/3973085752 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c0c005040 con 0x7f8c180f62c0 2024-01-18T06:49:44.299 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:44.298+0000 7f8c1cbfc700 1 --2- 172.21.15.194:0/3973085752 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c180f62c0 0x7f8c18106520 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f8c0c013510 tx=0x7f8c0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:49:44.299 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:44.298+0000 7f8c157fa700 1 -- 172.21.15.194:0/3973085752 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c0c009320 con 0x7f8c180f62c0 2024-01-18T06:49:44.299 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:44.298+0000 7f8c1dbfe700 1 -- 172.21.15.194:0/3973085752 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c18106c10 con 0x7f8c180f62c0 2024-01-18T06:49:44.299 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:44.298+0000 7f8c157fa700 1 -- 172.21.15.194:0/3973085752 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8c0c013f00 con 0x7f8c180f62c0 2024-01-18T06:49:44.299 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:44.298+0000 7f8c157fa700 1 -- 172.21.15.194:0/3973085752 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c0c014dc0 con 0x7f8c180f62c0 2024-01-18T06:49:44.299 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:44.298+0000 7f8c1dbfe700 1 -- 172.21.15.194:0/3973085752 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c181070a0 con 0x7f8c180f62c0 2024-01-18T06:49:44.299 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:44.299+0000 7f8c157fa700 1 -- 172.21.15.194:0/3973085752 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8c0c0095f0 con 0x7f8c180f62c0 2024-01-18T06:49:44.300 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:44.299+0000 7f8c1dbfe700 1 -- 172.21.15.194:0/3973085752 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c18045ad0 con 0x7f8c180f62c0 2024-01-18T06:49:44.300 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:44.299+0000 7f8c157fa700 1 --2- 172.21.15.194:0/3973085752 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f8c0802ff60 0x7f8c08032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:49:44.300 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:44.299+0000 7f8c157fa700 1 -- 172.21.15.194:0/3973085752 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8c0c041100 con 0x7f8c180f62c0 2024-01-18T06:49:44.300 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:44.299+0000 7f8c17fff700 1 --2- 172.21.15.194:0/3973085752 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f8c0802ff60 0x7f8c08032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:49:44.300 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:44.300+0000 7f8c17fff700 1 --2- 172.21.15.194:0/3973085752 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f8c0802ff60 0x7f8c08032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f8c040096a0 tx=0x7f8c04006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:49:44.303 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:44.304+0000 7f8c157fa700 1 -- 172.21.15.194:0/3973085752 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8c0c014420 con 0x7f8c180f62c0 2024-01-18T06:49:44.509 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:44.510+0000 7f8c1dbfe700 1 -- 172.21.15.194:0/3973085752 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8c18107f50 con 0x7f8c180f62c0 2024-01-18T06:49:44.510 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:44.510+0000 7f8c157fa700 1 -- 172.21.15.194:0/3973085752 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8c0c010230 con 0x7f8c180f62c0 2024-01-18T06:49:44.510 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:49:44.510 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:49:44.511 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:44.512+0000 7f8c1dbfe700 1 -- 172.21.15.194:0/3973085752 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f8c0802ff60 msgr2=0x7f8c08032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:44.511 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:44.512+0000 7f8c1dbfe700 1 --2- 172.21.15.194:0/3973085752 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f8c0802ff60 0x7f8c08032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f8c040096a0 tx=0x7f8c04006b40).stop 2024-01-18T06:49:44.511 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:44.512+0000 7f8c1dbfe700 1 -- 172.21.15.194:0/3973085752 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c180f62c0 msgr2=0x7f8c18106520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:44.511 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:44.512+0000 7f8c1dbfe700 1 --2- 172.21.15.194:0/3973085752 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c180f62c0 0x7f8c18106520 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f8c0c013510 tx=0x7f8c0c00b040).stop 2024-01-18T06:49:44.512 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:44.512+0000 7f8c1dbfe700 1 -- 172.21.15.194:0/3973085752 shutdown_connections 2024-01-18T06:49:44.512 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:44.512+0000 7f8c1dbfe700 1 --2- 172.21.15.194:0/3973085752 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f8c0802ff60 0x7f8c08032420 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:44.512 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:44.512+0000 7f8c1dbfe700 1 --2- 172.21.15.194:0/3973085752 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c180f62c0 0x7f8c18106520 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:44.512 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:44.512+0000 7f8c1dbfe700 1 -- 172.21.15.194:0/3973085752 >> 172.21.15.194:0/3973085752 conn(0x7f8c180f1ac0 msgr2=0x7f8c180f2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:49:44.512 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:44.512+0000 7f8c1dbfe700 1 -- 172.21.15.194:0/3973085752 shutdown_connections 2024-01-18T06:49:44.512 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:44.512+0000 7f8c1dbfe700 1 -- 172.21.15.194:0/3973085752 wait complete. 2024-01-18T06:49:44.512 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:49:45.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:45 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:49:43.983684+0000 mgr.smithi047.fuhzzr (mgr.14162) 38 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:49:45.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:45 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:49:44.511236+0000 mon.smithi047 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.194:0/3973085752' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:49:45.918 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:49:45.919 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:49:47.116 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:47.116+0000 7fe5f49dc700 1 -- 172.21.15.194:0/3620361221 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe5f00f51a0 msgr2=0x7fe5f00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:47.116 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:47.116+0000 7fe5f49dc700 1 --2- 172.21.15.194:0/3620361221 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe5f00f51a0 0x7fe5f00f55c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fe5e0009fb0 tx=0x7fe5e0009b30).stop 2024-01-18T06:49:47.117 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:47.116+0000 7fe5f49dc700 1 -- 172.21.15.194:0/3620361221 shutdown_connections 2024-01-18T06:49:47.117 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:47.116+0000 7fe5f49dc700 1 --2- 172.21.15.194:0/3620361221 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe5f00f51a0 0x7fe5f00f55c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:47.117 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:47.116+0000 7fe5f49dc700 1 -- 172.21.15.194:0/3620361221 >> 172.21.15.194:0/3620361221 conn(0x7fe5f00f09a0 msgr2=0x7fe5f00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:49:47.117 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:47.116+0000 7fe5f49dc700 1 -- 172.21.15.194:0/3620361221 shutdown_connections 2024-01-18T06:49:47.117 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:47.116+0000 7fe5f49dc700 1 -- 172.21.15.194:0/3620361221 wait complete. 2024-01-18T06:49:47.118 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:47.117+0000 7fe5f49dc700 1 Processor -- start 2024-01-18T06:49:47.118 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:47.117+0000 7fe5f49dc700 1 -- start start 2024-01-18T06:49:47.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:47.118+0000 7fe5f49dc700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe5f00f51a0 0x7fe5f01091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:49:47.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:47.118+0000 7fe5f49dc700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f0060240 con 0x7fe5f00f51a0 2024-01-18T06:49:47.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:47.118+0000 7fe5ef7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe5f00f51a0 0x7fe5f01091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:49:47.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:47.118+0000 7fe5ef7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe5f00f51a0 0x7fe5f01091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:49732/0 (socket says 172.21.15.194:49732) 2024-01-18T06:49:47.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:47.118+0000 7fe5ef7fe700 1 -- 172.21.15.194:0/3713658993 learned_addr learned my addr 172.21.15.194:0/3713658993 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:49:47.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:47.118+0000 7fe5ef7fe700 1 -- 172.21.15.194:0/3713658993 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e0005040 con 0x7fe5f00f51a0 2024-01-18T06:49:47.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:47.118+0000 7fe5ef7fe700 1 --2- 172.21.15.194:0/3713658993 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe5f00f51a0 0x7fe5f01091b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fe5e0013510 tx=0x7fe5e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:49:47.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:47.119+0000 7fe5d7fff700 1 -- 172.21.15.194:0/3713658993 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5e0009320 con 0x7fe5f00f51a0 2024-01-18T06:49:47.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:47.119+0000 7fe5f49dc700 1 -- 172.21.15.194:0/3713658993 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5f0060440 con 0x7fe5f00f51a0 2024-01-18T06:49:47.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:47.119+0000 7fe5d7fff700 1 -- 172.21.15.194:0/3713658993 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe5e0013f00 con 0x7fe5f00f51a0 2024-01-18T06:49:47.120 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:47.119+0000 7fe5d7fff700 1 -- 172.21.15.194:0/3713658993 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5e0014dc0 con 0x7fe5f00f51a0 2024-01-18T06:49:47.120 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:47.119+0000 7fe5f49dc700 1 -- 172.21.15.194:0/3713658993 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5f0109860 con 0x7fe5f00f51a0 2024-01-18T06:49:47.121 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:47.119+0000 7fe5d7fff700 1 -- 172.21.15.194:0/3713658993 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe5e0014420 con 0x7fe5f00f51a0 2024-01-18T06:49:47.121 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:47.120+0000 7fe5f49dc700 1 -- 172.21.15.194:0/3713658993 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5f0045ad0 con 0x7fe5f00f51a0 2024-01-18T06:49:47.121 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:47.120+0000 7fe5d7fff700 1 --2- 172.21.15.194:0/3713658993 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fe5d802ff60 0x7fe5d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:49:47.121 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:47.120+0000 7fe5d7fff700 1 -- 172.21.15.194:0/3713658993 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe5e00410d0 con 0x7fe5f00f51a0 2024-01-18T06:49:47.121 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:47.120+0000 7fe5eeffd700 1 --2- 172.21.15.194:0/3713658993 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fe5d802ff60 0x7fe5d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:49:47.121 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:47.120+0000 7fe5eeffd700 1 --2- 172.21.15.194:0/3713658993 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fe5d802ff60 0x7fe5d8032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fe5e40096a0 tx=0x7fe5e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:49:47.124 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:47.124+0000 7fe5d7fff700 1 -- 172.21.15.194:0/3713658993 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe5e00095f0 con 0x7fe5f00f51a0 2024-01-18T06:49:47.331 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:47.332+0000 7fe5f49dc700 1 -- 172.21.15.194:0/3713658993 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe5f010a080 con 0x7fe5f00f51a0 2024-01-18T06:49:47.332 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:47.332+0000 7fe5d7fff700 1 -- 172.21.15.194:0/3713658993 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe5e001cb10 con 0x7fe5f00f51a0 2024-01-18T06:49:47.332 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:49:47.332 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:49:47.334 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:47.334+0000 7fe5f49dc700 1 -- 172.21.15.194:0/3713658993 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fe5d802ff60 msgr2=0x7fe5d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:47.334 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:47.334+0000 7fe5f49dc700 1 --2- 172.21.15.194:0/3713658993 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fe5d802ff60 0x7fe5d8032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fe5e40096a0 tx=0x7fe5e4006b40).stop 2024-01-18T06:49:47.334 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:47.334+0000 7fe5f49dc700 1 -- 172.21.15.194:0/3713658993 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe5f00f51a0 msgr2=0x7fe5f01091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:47.334 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:47.334+0000 7fe5f49dc700 1 --2- 172.21.15.194:0/3713658993 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe5f00f51a0 0x7fe5f01091b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fe5e0013510 tx=0x7fe5e000b040).stop 2024-01-18T06:49:47.334 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:47.334+0000 7fe5f49dc700 1 -- 172.21.15.194:0/3713658993 shutdown_connections 2024-01-18T06:49:47.334 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:47.334+0000 7fe5f49dc700 1 --2- 172.21.15.194:0/3713658993 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fe5d802ff60 0x7fe5d8032420 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:47.334 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:47.334+0000 7fe5f49dc700 1 --2- 172.21.15.194:0/3713658993 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe5f00f51a0 0x7fe5f01091b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:47.334 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:47.334+0000 7fe5f49dc700 1 -- 172.21.15.194:0/3713658993 >> 172.21.15.194:0/3713658993 conn(0x7fe5f00f09a0 msgr2=0x7fe5f00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:49:47.334 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:47.334+0000 7fe5f49dc700 1 -- 172.21.15.194:0/3713658993 shutdown_connections 2024-01-18T06:49:47.334 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:47.334+0000 7fe5f49dc700 1 -- 172.21.15.194:0/3713658993 wait complete. 2024-01-18T06:49:47.335 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:49:47.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:47 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:49:45.984045+0000 mgr.smithi047.fuhzzr (mgr.14162) 39 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:49:48.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:48 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:49:47.333239+0000 mon.smithi047 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.194:0/3713658993' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:49:48.719 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:49:48.720 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:49:49.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:49 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:49:47.984408+0000 mgr.smithi047.fuhzzr (mgr.14162) 40 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:49:49.892 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:49.890+0000 7fba1de66700 1 -- 172.21.15.194:0/1153963644 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fba180f47c0 msgr2=0x7fba180f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:49.892 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:49.890+0000 7fba1de66700 1 --2- 172.21.15.194:0/1153963644 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fba180f47c0 0x7fba180f4be0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fba08009fb0 tx=0x7fba08009b30).stop 2024-01-18T06:49:49.892 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:49.890+0000 7fba1de66700 1 -- 172.21.15.194:0/1153963644 shutdown_connections 2024-01-18T06:49:49.892 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:49.890+0000 7fba1de66700 1 --2- 172.21.15.194:0/1153963644 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fba180f47c0 0x7fba180f4be0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:49.892 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:49.890+0000 7fba1de66700 1 -- 172.21.15.194:0/1153963644 >> 172.21.15.194:0/1153963644 conn(0x7fba180f02e0 msgr2=0x7fba180f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:49:49.893 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:49.890+0000 7fba1de66700 1 -- 172.21.15.194:0/1153963644 shutdown_connections 2024-01-18T06:49:49.893 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:49.890+0000 7fba1de66700 1 -- 172.21.15.194:0/1153963644 wait complete. 2024-01-18T06:49:49.893 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:49.890+0000 7fba1de66700 1 Processor -- start 2024-01-18T06:49:49.893 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:49.890+0000 7fba1de66700 1 -- start start 2024-01-18T06:49:49.893 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:49.890+0000 7fba1de66700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fba180f47c0 0x7fba180f7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:49:49.893 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:49.891+0000 7fba1de66700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba180f57f0 con 0x7fba180f47c0 2024-01-18T06:49:49.893 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:49.891+0000 7fba177fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fba180f47c0 0x7fba180f7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:49:49.893 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:49.891+0000 7fba177fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fba180f47c0 0x7fba180f7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:46640/0 (socket says 172.21.15.194:46640) 2024-01-18T06:49:49.893 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:49.891+0000 7fba177fe700 1 -- 172.21.15.194:0/2359709301 learned_addr learned my addr 172.21.15.194:0/2359709301 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:49:49.893 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:49.891+0000 7fba177fe700 1 -- 172.21.15.194:0/2359709301 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba08005040 con 0x7fba180f47c0 2024-01-18T06:49:49.893 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:49.891+0000 7fba177fe700 1 --2- 172.21.15.194:0/2359709301 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fba180f47c0 0x7fba180f7370 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fba08013510 tx=0x7fba0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:49:49.893 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:49.891+0000 7fb9fffff700 1 -- 172.21.15.194:0/2359709301 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba08009320 con 0x7fba180f47c0 2024-01-18T06:49:49.893 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:49.891+0000 7fba1de66700 1 -- 172.21.15.194:0/2359709301 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba180f5990 con 0x7fba180f47c0 2024-01-18T06:49:49.893 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:49.891+0000 7fb9fffff700 1 -- 172.21.15.194:0/2359709301 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fba08013f00 con 0x7fba180f47c0 2024-01-18T06:49:49.893 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:49.891+0000 7fb9fffff700 1 -- 172.21.15.194:0/2359709301 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba08014d90 con 0x7fba180f47c0 2024-01-18T06:49:49.894 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:49.891+0000 7fba1de66700 1 -- 172.21.15.194:0/2359709301 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba180f5e20 con 0x7fba180f47c0 2024-01-18T06:49:49.894 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:49.892+0000 7fb9fffff700 1 -- 172.21.15.194:0/2359709301 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fba080095f0 con 0x7fba180f47c0 2024-01-18T06:49:49.895 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:49.892+0000 7fba1de66700 1 -- 172.21.15.194:0/2359709301 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba18059100 con 0x7fba180f47c0 2024-01-18T06:49:49.895 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:49.893+0000 7fb9fffff700 1 --2- 172.21.15.194:0/2359709301 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fba0002ff60 0x7fba00032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:49:49.895 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:49.893+0000 7fb9fffff700 1 -- 172.21.15.194:0/2359709301 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fba08040fe0 con 0x7fba180f47c0 2024-01-18T06:49:49.895 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:49.893+0000 7fba16ffd700 1 --2- 172.21.15.194:0/2359709301 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fba0002ff60 0x7fba00032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:49:49.895 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:49.893+0000 7fba16ffd700 1 --2- 172.21.15.194:0/2359709301 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fba0002ff60 0x7fba00032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fba0c0096a0 tx=0x7fba0c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:49:49.898 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:49.897+0000 7fb9fffff700 1 -- 172.21.15.194:0/2359709301 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fba0801d7f0 con 0x7fba180f47c0 2024-01-18T06:49:50.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:50.103+0000 7fba1de66700 1 -- 172.21.15.194:0/2359709301 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fba180fbf60 con 0x7fba180f47c0 2024-01-18T06:49:50.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:50.104+0000 7fb9fffff700 1 -- 172.21.15.194:0/2359709301 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fba08010230 con 0x7fba180f47c0 2024-01-18T06:49:50.104 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:49:50.104 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:49:50.106 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:50.105+0000 7fba1de66700 1 -- 172.21.15.194:0/2359709301 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fba0002ff60 msgr2=0x7fba00032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:50.106 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:50.105+0000 7fba1de66700 1 --2- 172.21.15.194:0/2359709301 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fba0002ff60 0x7fba00032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fba0c0096a0 tx=0x7fba0c006b40).stop 2024-01-18T06:49:50.106 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:50.105+0000 7fba1de66700 1 -- 172.21.15.194:0/2359709301 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fba180f47c0 msgr2=0x7fba180f7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:50.106 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:50.105+0000 7fba1de66700 1 --2- 172.21.15.194:0/2359709301 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fba180f47c0 0x7fba180f7370 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fba08013510 tx=0x7fba0800b040).stop 2024-01-18T06:49:50.106 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:50.105+0000 7fba1de66700 1 -- 172.21.15.194:0/2359709301 shutdown_connections 2024-01-18T06:49:50.106 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:50.105+0000 7fba1de66700 1 --2- 172.21.15.194:0/2359709301 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fba0002ff60 0x7fba00032420 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:50.106 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:50.105+0000 7fba1de66700 1 --2- 172.21.15.194:0/2359709301 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fba180f47c0 0x7fba180f7370 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:50.106 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:50.105+0000 7fba1de66700 1 -- 172.21.15.194:0/2359709301 >> 172.21.15.194:0/2359709301 conn(0x7fba180f02e0 msgr2=0x7fba180f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:49:50.106 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:50.105+0000 7fba1de66700 1 -- 172.21.15.194:0/2359709301 shutdown_connections 2024-01-18T06:49:50.106 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:50.105+0000 7fba1de66700 1 -- 172.21.15.194:0/2359709301 wait complete. 2024-01-18T06:49:50.107 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:49:51.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:51 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:49:49.984813+0000 mgr.smithi047.fuhzzr (mgr.14162) 41 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:49:51.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:51 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:49:50.105767+0000 mon.smithi047 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.194:0/2359709301' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:49:51.506 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:49:51.506 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:49:52.716 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:52.714+0000 7fa894baf700 1 -- 172.21.15.194:0/1823629495 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa890060280 msgr2=0x7fa8900f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:52.716 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:52.714+0000 7fa894baf700 1 --2- 172.21.15.194:0/1823629495 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa890060280 0x7fa8900f90f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fa880009fb0 tx=0x7fa880009b30).stop 2024-01-18T06:49:52.716 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:52.714+0000 7fa894baf700 1 -- 172.21.15.194:0/1823629495 shutdown_connections 2024-01-18T06:49:52.716 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:52.714+0000 7fa894baf700 1 --2- 172.21.15.194:0/1823629495 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa890060280 0x7fa8900f90f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:52.717 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:52.714+0000 7fa894baf700 1 -- 172.21.15.194:0/1823629495 >> 172.21.15.194:0/1823629495 conn(0x7fa8900f0b70 msgr2=0x7fa8900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:49:52.717 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:52.714+0000 7fa894baf700 1 -- 172.21.15.194:0/1823629495 shutdown_connections 2024-01-18T06:49:52.717 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:52.714+0000 7fa894baf700 1 -- 172.21.15.194:0/1823629495 wait complete. 2024-01-18T06:49:52.717 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:52.714+0000 7fa894baf700 1 Processor -- start 2024-01-18T06:49:52.717 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:52.715+0000 7fa894baf700 1 -- start start 2024-01-18T06:49:52.717 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:52.715+0000 7fa894baf700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa890060280 0x7fa8901066c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:49:52.717 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:52.715+0000 7fa894baf700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa890106bb0 con 0x7fa890060280 2024-01-18T06:49:52.717 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:52.715+0000 7fa88e59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa890060280 0x7fa8901066c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:49:52.717 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:52.715+0000 7fa88e59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa890060280 0x7fa8901066c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:46656/0 (socket says 172.21.15.194:46656) 2024-01-18T06:49:52.717 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:52.715+0000 7fa88e59c700 1 -- 172.21.15.194:0/3388272886 learned_addr learned my addr 172.21.15.194:0/3388272886 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:49:52.717 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:52.715+0000 7fa88e59c700 1 -- 172.21.15.194:0/3388272886 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa880005040 con 0x7fa890060280 2024-01-18T06:49:52.717 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:52.715+0000 7fa88e59c700 1 --2- 172.21.15.194:0/3388272886 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa890060280 0x7fa8901066c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fa880013510 tx=0x7fa88000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:49:52.717 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:52.716+0000 7fa87effd700 1 -- 172.21.15.194:0/3388272886 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa880009320 con 0x7fa890060280 2024-01-18T06:49:52.717 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:52.716+0000 7fa894baf700 1 -- 172.21.15.194:0/3388272886 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa890106db0 con 0x7fa890060280 2024-01-18T06:49:52.718 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:52.716+0000 7fa87effd700 1 -- 172.21.15.194:0/3388272886 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa880013f00 con 0x7fa890060280 2024-01-18T06:49:52.718 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:52.716+0000 7fa87effd700 1 -- 172.21.15.194:0/3388272886 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa880014dc0 con 0x7fa890060280 2024-01-18T06:49:52.718 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:52.716+0000 7fa894baf700 1 -- 172.21.15.194:0/3388272886 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa890107240 con 0x7fa890060280 2024-01-18T06:49:52.718 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:52.716+0000 7fa87effd700 1 -- 172.21.15.194:0/3388272886 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa8800095f0 con 0x7fa890060280 2024-01-18T06:49:52.718 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:52.716+0000 7fa894baf700 1 -- 172.21.15.194:0/3388272886 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa890045ad0 con 0x7fa890060280 2024-01-18T06:49:52.719 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:52.717+0000 7fa87effd700 1 --2- 172.21.15.194:0/3388272886 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa87802ff60 0x7fa878032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:49:52.719 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:52.717+0000 7fa87effd700 1 -- 172.21.15.194:0/3388272886 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa880041040 con 0x7fa890060280 2024-01-18T06:49:52.719 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:52.717+0000 7fa88dd9b700 1 --2- 172.21.15.194:0/3388272886 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa87802ff60 0x7fa878032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:49:52.719 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:52.717+0000 7fa88dd9b700 1 --2- 172.21.15.194:0/3388272886 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa87802ff60 0x7fa878032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fa8840096a0 tx=0x7fa884006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:49:52.722 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:52.721+0000 7fa87effd700 1 -- 172.21.15.194:0/3388272886 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa880014420 con 0x7fa890060280 2024-01-18T06:49:52.925 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:52.925+0000 7fa894baf700 1 -- 172.21.15.194:0/3388272886 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa8901080f0 con 0x7fa890060280 2024-01-18T06:49:52.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:52.925+0000 7fa87effd700 1 -- 172.21.15.194:0/3388272886 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa880010230 con 0x7fa890060280 2024-01-18T06:49:52.926 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:49:52.926 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:49:52.928 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:52.927+0000 7fa894baf700 1 -- 172.21.15.194:0/3388272886 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa87802ff60 msgr2=0x7fa878032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:52.928 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:52.927+0000 7fa894baf700 1 --2- 172.21.15.194:0/3388272886 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa87802ff60 0x7fa878032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fa8840096a0 tx=0x7fa884006b40).stop 2024-01-18T06:49:52.928 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:52.927+0000 7fa894baf700 1 -- 172.21.15.194:0/3388272886 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa890060280 msgr2=0x7fa8901066c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:52.928 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:52.927+0000 7fa894baf700 1 --2- 172.21.15.194:0/3388272886 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa890060280 0x7fa8901066c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fa880013510 tx=0x7fa88000b040).stop 2024-01-18T06:49:52.928 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:52.927+0000 7fa894baf700 1 -- 172.21.15.194:0/3388272886 shutdown_connections 2024-01-18T06:49:52.928 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:52.927+0000 7fa894baf700 1 --2- 172.21.15.194:0/3388272886 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa87802ff60 0x7fa878032420 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:52.928 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:52.927+0000 7fa894baf700 1 --2- 172.21.15.194:0/3388272886 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa890060280 0x7fa8901066c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:52.928 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:52.927+0000 7fa894baf700 1 -- 172.21.15.194:0/3388272886 >> 172.21.15.194:0/3388272886 conn(0x7fa8900f0b70 msgr2=0x7fa8900f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:49:52.928 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:52.927+0000 7fa894baf700 1 -- 172.21.15.194:0/3388272886 shutdown_connections 2024-01-18T06:49:52.928 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:52.927+0000 7fa894baf700 1 -- 172.21.15.194:0/3388272886 wait complete. 2024-01-18T06:49:52.929 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:49:53.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:53 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:49:51.985130+0000 mgr.smithi047.fuhzzr (mgr.14162) 42 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:49:53.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:53 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:49:52.927199+0000 mon.smithi047 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.194:0/3388272886' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:49:54.326 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:49:54.327 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:49:55.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:55 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:49:53.985595+0000 mgr.smithi047.fuhzzr (mgr.14162) 43 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:49:55.568 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:55.566+0000 7fcb00a06700 1 -- 172.21.15.194:0/243708435 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcafc0f69d0 msgr2=0x7fcafc0f6df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:55.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:55.566+0000 7fcb00a06700 1 --2- 172.21.15.194:0/243708435 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcafc0f69d0 0x7fcafc0f6df0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fcaf0009fb0 tx=0x7fcaf0009b30).stop 2024-01-18T06:49:55.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:55.566+0000 7fcb00a06700 1 -- 172.21.15.194:0/243708435 shutdown_connections 2024-01-18T06:49:55.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:55.566+0000 7fcb00a06700 1 --2- 172.21.15.194:0/243708435 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcafc0f69d0 0x7fcafc0f6df0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:55.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:55.566+0000 7fcb00a06700 1 -- 172.21.15.194:0/243708435 >> 172.21.15.194:0/243708435 conn(0x7fcafc0f02e0 msgr2=0x7fcafc0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:49:55.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:55.566+0000 7fcb00a06700 1 -- 172.21.15.194:0/243708435 shutdown_connections 2024-01-18T06:49:55.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:55.566+0000 7fcb00a06700 1 -- 172.21.15.194:0/243708435 wait complete. 2024-01-18T06:49:55.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:55.567+0000 7fcb00a06700 1 Processor -- start 2024-01-18T06:49:55.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:55.567+0000 7fcb00a06700 1 -- start start 2024-01-18T06:49:55.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:55.567+0000 7fcb00a06700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcafc0f69d0 0x7fcafc1066c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:49:55.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:55.567+0000 7fcb00a06700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcafc106bb0 con 0x7fcafc0f69d0 2024-01-18T06:49:55.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:55.567+0000 7fcafa59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcafc0f69d0 0x7fcafc1066c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:49:55.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:55.567+0000 7fcafa59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcafc0f69d0 0x7fcafc1066c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:46682/0 (socket says 172.21.15.194:46682) 2024-01-18T06:49:55.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:55.567+0000 7fcafa59c700 1 -- 172.21.15.194:0/3756752950 learned_addr learned my addr 172.21.15.194:0/3756752950 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:49:55.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:55.568+0000 7fcafa59c700 1 -- 172.21.15.194:0/3756752950 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf0005040 con 0x7fcafc0f69d0 2024-01-18T06:49:55.570 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:55.568+0000 7fcafa59c700 1 --2- 172.21.15.194:0/3756752950 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcafc0f69d0 0x7fcafc1066c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fcaf0013510 tx=0x7fcaf000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:49:55.570 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:55.568+0000 7fcaeaffd700 1 -- 172.21.15.194:0/3756752950 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcaf0009320 con 0x7fcafc0f69d0 2024-01-18T06:49:55.570 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:55.568+0000 7fcb00a06700 1 -- 172.21.15.194:0/3756752950 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcafc106db0 con 0x7fcafc0f69d0 2024-01-18T06:49:55.570 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:55.568+0000 7fcaeaffd700 1 -- 172.21.15.194:0/3756752950 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcaf0013f00 con 0x7fcafc0f69d0 2024-01-18T06:49:55.570 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:55.568+0000 7fcaeaffd700 1 -- 172.21.15.194:0/3756752950 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcaf0014d90 con 0x7fcafc0f69d0 2024-01-18T06:49:55.570 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:55.568+0000 7fcb00a06700 1 -- 172.21.15.194:0/3756752950 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcafc107240 con 0x7fcafc0f69d0 2024-01-18T06:49:55.571 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:55.569+0000 7fcb00a06700 1 -- 172.21.15.194:0/3756752950 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcafc045ad0 con 0x7fcafc0f69d0 2024-01-18T06:49:55.571 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:55.569+0000 7fcaeaffd700 1 -- 172.21.15.194:0/3756752950 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcaf00263a0 con 0x7fcafc0f69d0 2024-01-18T06:49:55.571 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:55.569+0000 7fcaeaffd700 1 --2- 172.21.15.194:0/3756752950 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fcae402ff60 0x7fcae4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:49:55.571 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:55.569+0000 7fcaeaffd700 1 -- 172.21.15.194:0/3756752950 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcaf0041080 con 0x7fcafc0f69d0 2024-01-18T06:49:55.571 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:55.569+0000 7fcaf9d9b700 1 --2- 172.21.15.194:0/3756752950 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fcae402ff60 0x7fcae4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:49:55.571 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:55.570+0000 7fcaf9d9b700 1 --2- 172.21.15.194:0/3756752950 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fcae402ff60 0x7fcae4032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fcaec0096a0 tx=0x7fcaec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:49:55.574 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:55.574+0000 7fcaeaffd700 1 -- 172.21.15.194:0/3756752950 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcaf00095f0 con 0x7fcafc0f69d0 2024-01-18T06:49:55.778 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:55.778+0000 7fcb00a06700 1 -- 172.21.15.194:0/3756752950 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcafc1080f0 con 0x7fcafc0f69d0 2024-01-18T06:49:55.779 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:55.778+0000 7fcaeaffd700 1 -- 172.21.15.194:0/3756752950 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcaf001c970 con 0x7fcafc0f69d0 2024-01-18T06:49:55.779 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:49:55.779 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:49:55.781 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:55.780+0000 7fcb00a06700 1 -- 172.21.15.194:0/3756752950 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fcae402ff60 msgr2=0x7fcae4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:55.781 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:55.780+0000 7fcb00a06700 1 --2- 172.21.15.194:0/3756752950 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fcae402ff60 0x7fcae4032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fcaec0096a0 tx=0x7fcaec006b40).stop 2024-01-18T06:49:55.781 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:55.780+0000 7fcb00a06700 1 -- 172.21.15.194:0/3756752950 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcafc0f69d0 msgr2=0x7fcafc1066c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:55.781 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:55.780+0000 7fcb00a06700 1 --2- 172.21.15.194:0/3756752950 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcafc0f69d0 0x7fcafc1066c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fcaf0013510 tx=0x7fcaf000b040).stop 2024-01-18T06:49:55.781 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:55.780+0000 7fcb00a06700 1 -- 172.21.15.194:0/3756752950 shutdown_connections 2024-01-18T06:49:55.781 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:55.780+0000 7fcb00a06700 1 --2- 172.21.15.194:0/3756752950 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fcae402ff60 0x7fcae4032420 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:55.781 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:55.780+0000 7fcb00a06700 1 --2- 172.21.15.194:0/3756752950 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcafc0f69d0 0x7fcafc1066c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:55.781 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:55.780+0000 7fcb00a06700 1 -- 172.21.15.194:0/3756752950 >> 172.21.15.194:0/3756752950 conn(0x7fcafc0f02e0 msgr2=0x7fcafc0f4e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:49:55.781 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:55.780+0000 7fcb00a06700 1 -- 172.21.15.194:0/3756752950 shutdown_connections 2024-01-18T06:49:55.781 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:55.780+0000 7fcb00a06700 1 -- 172.21.15.194:0/3756752950 wait complete. 2024-01-18T06:49:55.782 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:49:56.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:49:55.780238+0000 mon.smithi047 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.194:0/3756752950' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:49:57.173 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:49:57.174 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:49:57.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:57 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:49:55.985930+0000 mgr.smithi047.fuhzzr (mgr.14162) 44 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:49:58.432 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:58.432+0000 7f51ab356700 1 -- 172.21.15.194:0/184475200 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f51a40f5410 msgr2=0x7f51a40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:58.432 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:58.432+0000 7f51ab356700 1 --2- 172.21.15.194:0/184475200 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f51a40f5410 0x7f51a40f5830 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f5194009fb0 tx=0x7f5194009b30).stop 2024-01-18T06:49:58.432 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:58.432+0000 7f51ab356700 1 -- 172.21.15.194:0/184475200 shutdown_connections 2024-01-18T06:49:58.432 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:58.432+0000 7f51ab356700 1 --2- 172.21.15.194:0/184475200 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f51a40f5410 0x7f51a40f5830 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:58.432 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:58.432+0000 7f51ab356700 1 -- 172.21.15.194:0/184475200 >> 172.21.15.194:0/184475200 conn(0x7f51a40f0b70 msgr2=0x7f51a40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:49:58.433 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:58.432+0000 7f51ab356700 1 -- 172.21.15.194:0/184475200 shutdown_connections 2024-01-18T06:49:58.433 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:58.432+0000 7f51ab356700 1 -- 172.21.15.194:0/184475200 wait complete. 2024-01-18T06:49:58.433 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:58.432+0000 7f51ab356700 1 Processor -- start 2024-01-18T06:49:58.433 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:58.433+0000 7f51ab356700 1 -- start start 2024-01-18T06:49:58.433 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:58.433+0000 7f51ab356700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f51a40f5410 0x7f51a4108970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:49:58.433 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:58.433+0000 7f51ab356700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51a4108e60 con 0x7f51a40f5410 2024-01-18T06:49:58.434 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:58.433+0000 7f51a90f2700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f51a40f5410 0x7f51a4108970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:49:58.434 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:58.433+0000 7f51a90f2700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f51a40f5410 0x7f51a4108970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:49810/0 (socket says 172.21.15.194:49810) 2024-01-18T06:49:58.435 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:58.433+0000 7f51a90f2700 1 -- 172.21.15.194:0/1027472093 learned_addr learned my addr 172.21.15.194:0/1027472093 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:49:58.435 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:58.433+0000 7f51a90f2700 1 -- 172.21.15.194:0/1027472093 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5194005040 con 0x7f51a40f5410 2024-01-18T06:49:58.435 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:58.434+0000 7f51a90f2700 1 --2- 172.21.15.194:0/1027472093 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f51a40f5410 0x7f51a4108970 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f5194009f80 tx=0x7f519400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:49:58.435 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:58.434+0000 7f5199ffb700 1 -- 172.21.15.194:0/1027472093 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5194009320 con 0x7f51a40f5410 2024-01-18T06:49:58.435 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:58.434+0000 7f51ab356700 1 -- 172.21.15.194:0/1027472093 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51a4109060 con 0x7f51a40f5410 2024-01-18T06:49:58.435 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:58.434+0000 7f5199ffb700 1 -- 172.21.15.194:0/1027472093 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5194013ec0 con 0x7f51a40f5410 2024-01-18T06:49:58.435 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:58.434+0000 7f5199ffb700 1 -- 172.21.15.194:0/1027472093 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f519401ece0 con 0x7f51a40f5410 2024-01-18T06:49:58.435 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:58.434+0000 7f51ab356700 1 -- 172.21.15.194:0/1027472093 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51a41094f0 con 0x7f51a40f5410 2024-01-18T06:49:58.435 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:58.435+0000 7f5199ffb700 1 -- 172.21.15.194:0/1027472093 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f519401e420 con 0x7f51a40f5410 2024-01-18T06:49:58.435 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:58.435+0000 7f51ab356700 1 -- 172.21.15.194:0/1027472093 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51a4045ad0 con 0x7f51a40f5410 2024-01-18T06:49:58.435 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:58.435+0000 7f5199ffb700 1 --2- 172.21.15.194:0/1027472093 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f519002ff60 0x7f5190032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:49:58.435 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:58.435+0000 7f5199ffb700 1 -- 172.21.15.194:0/1027472093 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5194040e40 con 0x7f51a40f5410 2024-01-18T06:49:58.435 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:58.435+0000 7f51a88f1700 1 --2- 172.21.15.194:0/1027472093 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f519002ff60 0x7f5190032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:49:58.436 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:58.435+0000 7f51a88f1700 1 --2- 172.21.15.194:0/1027472093 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f519002ff60 0x7f5190032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f51a00096a0 tx=0x7f51a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:49:58.439 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:58.440+0000 7f5199ffb700 1 -- 172.21.15.194:0/1027472093 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f519401e730 con 0x7f51a40f5410 2024-01-18T06:49:58.644 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:58.644+0000 7f51ab356700 1 -- 172.21.15.194:0/1027472093 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f51a410a340 con 0x7f51a40f5410 2024-01-18T06:49:58.644 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:58.644+0000 7f5199ffb700 1 -- 172.21.15.194:0/1027472093 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f519401cb10 con 0x7f51a40f5410 2024-01-18T06:49:58.644 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:49:58.645 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:49:58.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:58.646+0000 7f51ab356700 1 -- 172.21.15.194:0/1027472093 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f519002ff60 msgr2=0x7f5190032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:58.646 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:58.646+0000 7f51ab356700 1 --2- 172.21.15.194:0/1027472093 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f519002ff60 0x7f5190032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f51a00096a0 tx=0x7f51a0006b40).stop 2024-01-18T06:49:58.646 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:58.646+0000 7f51ab356700 1 -- 172.21.15.194:0/1027472093 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f51a40f5410 msgr2=0x7f51a4108970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:49:58.646 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:58.646+0000 7f51ab356700 1 --2- 172.21.15.194:0/1027472093 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f51a40f5410 0x7f51a4108970 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f5194009f80 tx=0x7f519400b040).stop 2024-01-18T06:49:58.646 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:58.646+0000 7f51ab356700 1 -- 172.21.15.194:0/1027472093 shutdown_connections 2024-01-18T06:49:58.646 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:58.646+0000 7f51ab356700 1 --2- 172.21.15.194:0/1027472093 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f519002ff60 0x7f5190032420 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:58.646 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:58.646+0000 7f51ab356700 1 --2- 172.21.15.194:0/1027472093 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f51a40f5410 0x7f51a4108970 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:49:58.646 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:58.646+0000 7f51ab356700 1 -- 172.21.15.194:0/1027472093 >> 172.21.15.194:0/1027472093 conn(0x7f51a40f0b70 msgr2=0x7f51a405fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:49:58.646 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:58.646+0000 7f51ab356700 1 -- 172.21.15.194:0/1027472093 shutdown_connections 2024-01-18T06:49:58.646 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:49:58.646+0000 7f51ab356700 1 -- 172.21.15.194:0/1027472093 wait complete. 2024-01-18T06:49:58.646 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:49:59.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:58 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:49:57.986293+0000 mgr.smithi047.fuhzzr (mgr.14162) 45 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:49:59.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:49:58 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:49:58.645188+0000 mon.smithi047 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.194:0/1027472093' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:50:00.043 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:50:00.043 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:50:01.282 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:01.278+0000 7f60002a9700 1 -- 172.21.15.194:0/3066136103 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5ff80f6510 msgr2=0x7f5ff80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:01.282 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:01.278+0000 7f60002a9700 1 --2- 172.21.15.194:0/3066136103 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5ff80f6510 0x7f5ff80f6930 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f5ff4009fb0 tx=0x7f5ff4009b30).stop 2024-01-18T06:50:01.283 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:01.278+0000 7f60002a9700 1 -- 172.21.15.194:0/3066136103 shutdown_connections 2024-01-18T06:50:01.283 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:01.278+0000 7f60002a9700 1 --2- 172.21.15.194:0/3066136103 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5ff80f6510 0x7f5ff80f6930 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:01.283 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:01.278+0000 7f60002a9700 1 -- 172.21.15.194:0/3066136103 >> 172.21.15.194:0/3066136103 conn(0x7f5ff80f1c70 msgr2=0x7f5ff80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:50:01.283 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:01.279+0000 7f60002a9700 1 -- 172.21.15.194:0/3066136103 shutdown_connections 2024-01-18T06:50:01.283 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:01.279+0000 7f60002a9700 1 -- 172.21.15.194:0/3066136103 wait complete. 2024-01-18T06:50:01.283 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:01.279+0000 7f60002a9700 1 Processor -- start 2024-01-18T06:50:01.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:01.279+0000 7f60002a9700 1 -- start start 2024-01-18T06:50:01.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:01.279+0000 7f60002a9700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5ff80f6510 0x7f5ff8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:50:01.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:01.279+0000 7f60002a9700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff8106c20 con 0x7f5ff80f6510 2024-01-18T06:50:01.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:01.280+0000 7f5ffe045700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5ff80f6510 0x7f5ff8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:50:01.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:01.280+0000 7f5ffe045700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5ff80f6510 0x7f5ff8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:49828/0 (socket says 172.21.15.194:49828) 2024-01-18T06:50:01.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:01.280+0000 7f5ffe045700 1 -- 172.21.15.194:0/1528796479 learned_addr learned my addr 172.21.15.194:0/1528796479 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:50:01.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:01.280+0000 7f5ffe045700 1 -- 172.21.15.194:0/1528796479 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ff4005040 con 0x7f5ff80f6510 2024-01-18T06:50:01.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:01.280+0000 7f5ffe045700 1 --2- 172.21.15.194:0/1528796479 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5ff80f6510 0x7f5ff8106730 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f5ff4013510 tx=0x7f5ff400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:50:01.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:01.280+0000 7f5feeffd700 1 -- 172.21.15.194:0/1528796479 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ff4009320 con 0x7f5ff80f6510 2024-01-18T06:50:01.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:01.280+0000 7f60002a9700 1 -- 172.21.15.194:0/1528796479 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ff8106e20 con 0x7f5ff80f6510 2024-01-18T06:50:01.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:01.280+0000 7f5feeffd700 1 -- 172.21.15.194:0/1528796479 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5ff4013f00 con 0x7f5ff80f6510 2024-01-18T06:50:01.286 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:01.280+0000 7f5feeffd700 1 -- 172.21.15.194:0/1528796479 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ff4014d90 con 0x7f5ff80f6510 2024-01-18T06:50:01.286 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:01.280+0000 7f60002a9700 1 -- 172.21.15.194:0/1528796479 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ff81072b0 con 0x7f5ff80f6510 2024-01-18T06:50:01.286 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:01.281+0000 7f5feeffd700 1 -- 172.21.15.194:0/1528796479 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5ff4014420 con 0x7f5ff80f6510 2024-01-18T06:50:01.286 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:01.281+0000 7f60002a9700 1 -- 172.21.15.194:0/1528796479 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ff8045ad0 con 0x7f5ff80f6510 2024-01-18T06:50:01.286 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:01.281+0000 7f5feeffd700 1 --2- 172.21.15.194:0/1528796479 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f5fe402ff60 0x7f5fe4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:50:01.286 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:01.281+0000 7f5feeffd700 1 -- 172.21.15.194:0/1528796479 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5ff4041000 con 0x7f5ff80f6510 2024-01-18T06:50:01.287 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:01.282+0000 7f5ffd844700 1 --2- 172.21.15.194:0/1528796479 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f5fe402ff60 0x7f5fe4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:50:01.287 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:01.282+0000 7f5ffd844700 1 --2- 172.21.15.194:0/1528796479 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f5fe402ff60 0x7f5fe4032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f5fe80096a0 tx=0x7f5fe8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:50:01.287 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:01.286+0000 7f5feeffd700 1 -- 172.21.15.194:0/1528796479 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5ff40095f0 con 0x7f5ff80f6510 2024-01-18T06:50:01.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:01 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:49:59.986708+0000 mgr.smithi047.fuhzzr (mgr.14162) 46 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:50:01.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:01.491+0000 7f60002a9700 1 -- 172.21.15.194:0/1528796479 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5ff80fbf60 con 0x7f5ff80f6510 2024-01-18T06:50:01.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:01.492+0000 7f5feeffd700 1 -- 172.21.15.194:0/1528796479 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5ff401db70 con 0x7f5ff80f6510 2024-01-18T06:50:01.492 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:50:01.492 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:50:01.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:01.494+0000 7f60002a9700 1 -- 172.21.15.194:0/1528796479 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f5fe402ff60 msgr2=0x7f5fe4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:01.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:01.494+0000 7f60002a9700 1 --2- 172.21.15.194:0/1528796479 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f5fe402ff60 0x7f5fe4032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f5fe80096a0 tx=0x7f5fe8006b40).stop 2024-01-18T06:50:01.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:01.494+0000 7f60002a9700 1 -- 172.21.15.194:0/1528796479 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5ff80f6510 msgr2=0x7f5ff8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:01.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:01.494+0000 7f60002a9700 1 --2- 172.21.15.194:0/1528796479 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5ff80f6510 0x7f5ff8106730 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f5ff4013510 tx=0x7f5ff400b040).stop 2024-01-18T06:50:01.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:01.494+0000 7f60002a9700 1 -- 172.21.15.194:0/1528796479 shutdown_connections 2024-01-18T06:50:01.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:01.494+0000 7f60002a9700 1 --2- 172.21.15.194:0/1528796479 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f5fe402ff60 0x7f5fe4032420 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:01.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:01.494+0000 7f60002a9700 1 --2- 172.21.15.194:0/1528796479 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5ff80f6510 0x7f5ff8106730 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:01.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:01.494+0000 7f60002a9700 1 -- 172.21.15.194:0/1528796479 >> 172.21.15.194:0/1528796479 conn(0x7f5ff80f1c70 msgr2=0x7f5ff80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:50:01.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:01.494+0000 7f60002a9700 1 -- 172.21.15.194:0/1528796479 shutdown_connections 2024-01-18T06:50:01.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:01.494+0000 7f60002a9700 1 -- 172.21.15.194:0/1528796479 wait complete. 2024-01-18T06:50:01.494 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:50:02.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:02 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:50:01.493036+0000 mon.smithi047 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.194:0/1528796479' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:50:02.902 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:50:02.902 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:50:03.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:03 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:50:01.987041+0000 mgr.smithi047.fuhzzr (mgr.14162) 47 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:50:04.112 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:04.110+0000 7f5cea2be700 1 -- 172.21.15.194:0/1613183552 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5ce40f51a0 msgr2=0x7f5ce40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:04.112 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:04.110+0000 7f5cea2be700 1 --2- 172.21.15.194:0/1613183552 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5ce40f51a0 0x7f5ce40f55c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f5cd4009fb0 tx=0x7f5cd4009b30).stop 2024-01-18T06:50:04.112 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:04.110+0000 7f5cea2be700 1 -- 172.21.15.194:0/1613183552 shutdown_connections 2024-01-18T06:50:04.112 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:04.110+0000 7f5cea2be700 1 --2- 172.21.15.194:0/1613183552 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5ce40f51a0 0x7f5ce40f55c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:04.113 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:04.110+0000 7f5cea2be700 1 -- 172.21.15.194:0/1613183552 >> 172.21.15.194:0/1613183552 conn(0x7f5ce40f09a0 msgr2=0x7f5ce40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:50:04.113 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:04.111+0000 7f5cea2be700 1 -- 172.21.15.194:0/1613183552 shutdown_connections 2024-01-18T06:50:04.113 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:04.111+0000 7f5cea2be700 1 -- 172.21.15.194:0/1613183552 wait complete. 2024-01-18T06:50:04.113 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:04.111+0000 7f5cea2be700 1 Processor -- start 2024-01-18T06:50:04.113 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:04.111+0000 7f5cea2be700 1 -- start start 2024-01-18T06:50:04.113 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:04.111+0000 7f5cea2be700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5ce40f51a0 0x7f5ce41064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:50:04.113 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:04.111+0000 7f5cea2be700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce4106990 con 0x7f5ce40f51a0 2024-01-18T06:50:04.113 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:04.111+0000 7f5ce92bc700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5ce40f51a0 0x7f5ce41064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:50:04.113 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:04.112+0000 7f5ce92bc700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5ce40f51a0 0x7f5ce41064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:49846/0 (socket says 172.21.15.194:49846) 2024-01-18T06:50:04.113 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:04.112+0000 7f5ce92bc700 1 -- 172.21.15.194:0/3797267004 learned_addr learned my addr 172.21.15.194:0/3797267004 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:50:04.114 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:04.112+0000 7f5ce92bc700 1 -- 172.21.15.194:0/3797267004 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cd4005040 con 0x7f5ce40f51a0 2024-01-18T06:50:04.114 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:04.112+0000 7f5ce92bc700 1 --2- 172.21.15.194:0/3797267004 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5ce40f51a0 0x7f5ce41064a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f5cd4013510 tx=0x7f5cd400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:50:04.114 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:04.112+0000 7f5cd9ffb700 1 -- 172.21.15.194:0/3797267004 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cd4009320 con 0x7f5ce40f51a0 2024-01-18T06:50:04.114 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:04.112+0000 7f5cd9ffb700 1 -- 172.21.15.194:0/3797267004 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5cd4013f00 con 0x7f5ce40f51a0 2024-01-18T06:50:04.114 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:04.112+0000 7f5cea2be700 1 -- 172.21.15.194:0/3797267004 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ce4106b90 con 0x7f5ce40f51a0 2024-01-18T06:50:04.114 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:04.112+0000 7f5cd9ffb700 1 -- 172.21.15.194:0/3797267004 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cd4014dc0 con 0x7f5ce40f51a0 2024-01-18T06:50:04.114 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:04.112+0000 7f5cea2be700 1 -- 172.21.15.194:0/3797267004 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ce4107020 con 0x7f5ce40f51a0 2024-01-18T06:50:04.115 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:04.113+0000 7f5cd9ffb700 1 -- 172.21.15.194:0/3797267004 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5cd40095f0 con 0x7f5ce40f51a0 2024-01-18T06:50:04.115 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:04.113+0000 7f5cea2be700 1 -- 172.21.15.194:0/3797267004 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ce4045ad0 con 0x7f5ce40f51a0 2024-01-18T06:50:04.115 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:04.113+0000 7f5cd9ffb700 1 --2- 172.21.15.194:0/3797267004 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f5cd002ff60 0x7f5cd0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:50:04.115 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:04.113+0000 7f5cd9ffb700 1 -- 172.21.15.194:0/3797267004 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5cd4041100 con 0x7f5ce40f51a0 2024-01-18T06:50:04.116 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:04.114+0000 7f5ce8abb700 1 --2- 172.21.15.194:0/3797267004 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f5cd002ff60 0x7f5cd0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:50:04.116 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:04.114+0000 7f5ce8abb700 1 --2- 172.21.15.194:0/3797267004 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f5cd002ff60 0x7f5cd0032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f5ce00096a0 tx=0x7f5ce0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:50:04.117 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:04.118+0000 7f5cd9ffb700 1 -- 172.21.15.194:0/3797267004 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5cd4014420 con 0x7f5ce40f51a0 2024-01-18T06:50:04.326 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:04.326+0000 7f5cea2be700 1 -- 172.21.15.194:0/3797267004 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5ce4107ed0 con 0x7f5ce40f51a0 2024-01-18T06:50:04.326 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:04.327+0000 7f5cd9ffb700 1 -- 172.21.15.194:0/3797267004 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5cd4010230 con 0x7f5ce40f51a0 2024-01-18T06:50:04.326 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:50:04.327 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:50:04.328 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:04.328+0000 7f5cea2be700 1 -- 172.21.15.194:0/3797267004 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f5cd002ff60 msgr2=0x7f5cd0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:04.328 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:04.328+0000 7f5cea2be700 1 --2- 172.21.15.194:0/3797267004 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f5cd002ff60 0x7f5cd0032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f5ce00096a0 tx=0x7f5ce0006b40).stop 2024-01-18T06:50:04.328 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:04.328+0000 7f5cea2be700 1 -- 172.21.15.194:0/3797267004 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5ce40f51a0 msgr2=0x7f5ce41064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:04.328 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:04.328+0000 7f5cea2be700 1 --2- 172.21.15.194:0/3797267004 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5ce40f51a0 0x7f5ce41064a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f5cd4013510 tx=0x7f5cd400b040).stop 2024-01-18T06:50:04.328 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:04.328+0000 7f5cea2be700 1 -- 172.21.15.194:0/3797267004 shutdown_connections 2024-01-18T06:50:04.329 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:04.328+0000 7f5cea2be700 1 --2- 172.21.15.194:0/3797267004 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f5cd002ff60 0x7f5cd0032420 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:04.329 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:04.328+0000 7f5cea2be700 1 --2- 172.21.15.194:0/3797267004 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5ce40f51a0 0x7f5ce41064a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:04.329 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:04.328+0000 7f5cea2be700 1 -- 172.21.15.194:0/3797267004 >> 172.21.15.194:0/3797267004 conn(0x7f5ce40f09a0 msgr2=0x7f5ce405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:50:04.329 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:04.328+0000 7f5cea2be700 1 -- 172.21.15.194:0/3797267004 shutdown_connections 2024-01-18T06:50:04.329 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:04.328+0000 7f5cea2be700 1 -- 172.21.15.194:0/3797267004 wait complete. 2024-01-18T06:50:04.329 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:50:05.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:05 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:50:03.987452+0000 mgr.smithi047.fuhzzr (mgr.14162) 48 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:50:05.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:05 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:50:04.327712+0000 mon.smithi047 (mon.0) 141 : audit [DBG] from='client.? 172.21.15.194:0/3797267004' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:50:05.727 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:50:05.727 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:50:06.936 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:06.935+0000 7f23bf04c700 1 -- 172.21.15.194:0/1899581196 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23b80601e0 msgr2=0x7f23b80f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:06.937 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:06.935+0000 7f23bf04c700 1 --2- 172.21.15.194:0/1899581196 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23b80601e0 0x7f23b80f8f20 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f23b0009960 tx=0x7f23b0004d10).stop 2024-01-18T06:50:06.937 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:06.935+0000 7f23bf04c700 1 -- 172.21.15.194:0/1899581196 shutdown_connections 2024-01-18T06:50:06.937 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:06.935+0000 7f23bf04c700 1 --2- 172.21.15.194:0/1899581196 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23b80601e0 0x7f23b80f8f20 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:06.937 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:06.935+0000 7f23bf04c700 1 -- 172.21.15.194:0/1899581196 >> 172.21.15.194:0/1899581196 conn(0x7f23b80f09a0 msgr2=0x7f23b80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:50:06.937 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:06.935+0000 7f23bf04c700 1 -- 172.21.15.194:0/1899581196 shutdown_connections 2024-01-18T06:50:06.937 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:06.935+0000 7f23bf04c700 1 -- 172.21.15.194:0/1899581196 wait complete. 2024-01-18T06:50:06.937 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:06.936+0000 7f23bf04c700 1 Processor -- start 2024-01-18T06:50:06.937 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:06.936+0000 7f23bf04c700 1 -- start start 2024-01-18T06:50:06.937 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:06.936+0000 7f23bf04c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23b80601e0 0x7f23b80f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:50:06.937 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:06.936+0000 7f23bf04c700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b80f4e20 con 0x7f23b80601e0 2024-01-18T06:50:06.937 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:06.936+0000 7f23be04a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23b80601e0 0x7f23b80f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:50:06.937 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:06.936+0000 7f23be04a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23b80601e0 0x7f23b80f69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:49864/0 (socket says 172.21.15.194:49864) 2024-01-18T06:50:06.937 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:06.936+0000 7f23be04a700 1 -- 172.21.15.194:0/226719797 learned_addr learned my addr 172.21.15.194:0/226719797 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:50:06.938 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:06.937+0000 7f23be04a700 1 -- 172.21.15.194:0/226719797 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b0009640 con 0x7f23b80601e0 2024-01-18T06:50:06.938 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:06.937+0000 7f23be04a700 1 --2- 172.21.15.194:0/226719797 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23b80601e0 0x7f23b80f69a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f23b0004b70 tx=0x7f23b0004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:50:06.938 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:06.937+0000 7f23b6ffd700 1 -- 172.21.15.194:0/226719797 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23b0018070 con 0x7f23b80601e0 2024-01-18T06:50:06.938 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:06.937+0000 7f23bf04c700 1 -- 172.21.15.194:0/226719797 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23b80f4fc0 con 0x7f23b80601e0 2024-01-18T06:50:06.938 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:06.937+0000 7f23b6ffd700 1 -- 172.21.15.194:0/226719797 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f23b000b870 con 0x7f23b80601e0 2024-01-18T06:50:06.938 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:06.937+0000 7f23b6ffd700 1 -- 172.21.15.194:0/226719797 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23b001c560 con 0x7f23b80601e0 2024-01-18T06:50:06.938 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:06.937+0000 7f23bf04c700 1 -- 172.21.15.194:0/226719797 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23b80f5450 con 0x7f23b80601e0 2024-01-18T06:50:06.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:06.938+0000 7f23b6ffd700 1 -- 172.21.15.194:0/226719797 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f23b001c6c0 con 0x7f23b80601e0 2024-01-18T06:50:06.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:06.938+0000 7f23b6ffd700 1 --2- 172.21.15.194:0/226719797 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f23a002ffb0 0x7f23a0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:50:06.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:06.938+0000 7f23bf04c700 1 -- 172.21.15.194:0/226719797 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23b8059100 con 0x7f23b80601e0 2024-01-18T06:50:06.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:06.938+0000 7f23b6ffd700 1 -- 172.21.15.194:0/226719797 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f23b00415b0 con 0x7f23b80601e0 2024-01-18T06:50:06.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:06.938+0000 7f23bd849700 1 --2- 172.21.15.194:0/226719797 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f23a002ffb0 0x7f23a0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:50:06.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:06.939+0000 7f23bd849700 1 --2- 172.21.15.194:0/226719797 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f23a002ffb0 0x7f23a0032470 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f23ac0096a0 tx=0x7f23ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:50:06.942 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:06.942+0000 7f23b6ffd700 1 -- 172.21.15.194:0/226719797 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f23b0012c00 con 0x7f23b80601e0 2024-01-18T06:50:07.147 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:07.147+0000 7f23bf04c700 1 -- 172.21.15.194:0/226719797 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f23b80fbcd0 con 0x7f23b80601e0 2024-01-18T06:50:07.147 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:07.148+0000 7f23b6ffd700 1 -- 172.21.15.194:0/226719797 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f23b0004e10 con 0x7f23b80601e0 2024-01-18T06:50:07.147 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:50:07.148 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:50:07.149 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:07.149+0000 7f23bf04c700 1 -- 172.21.15.194:0/226719797 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f23a002ffb0 msgr2=0x7f23a0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:07.149 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:07.149+0000 7f23bf04c700 1 --2- 172.21.15.194:0/226719797 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f23a002ffb0 0x7f23a0032470 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f23ac0096a0 tx=0x7f23ac006b40).stop 2024-01-18T06:50:07.149 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:07.149+0000 7f23bf04c700 1 -- 172.21.15.194:0/226719797 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23b80601e0 msgr2=0x7f23b80f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:07.149 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:07.149+0000 7f23bf04c700 1 --2- 172.21.15.194:0/226719797 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23b80601e0 0x7f23b80f69a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f23b0004b70 tx=0x7f23b0004d10).stop 2024-01-18T06:50:07.149 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:07.149+0000 7f23bf04c700 1 -- 172.21.15.194:0/226719797 shutdown_connections 2024-01-18T06:50:07.149 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:07.149+0000 7f23bf04c700 1 --2- 172.21.15.194:0/226719797 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f23a002ffb0 0x7f23a0032470 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:07.149 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:07.149+0000 7f23bf04c700 1 --2- 172.21.15.194:0/226719797 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23b80601e0 0x7f23b80f69a0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:07.149 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:07.149+0000 7f23bf04c700 1 -- 172.21.15.194:0/226719797 >> 172.21.15.194:0/226719797 conn(0x7f23b80f09a0 msgr2=0x7f23b80f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:50:07.150 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:07.149+0000 7f23bf04c700 1 -- 172.21.15.194:0/226719797 shutdown_connections 2024-01-18T06:50:07.150 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:07.150+0000 7f23bf04c700 1 -- 172.21.15.194:0/226719797 wait complete. 2024-01-18T06:50:07.150 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:50:07.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:07 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:50:05.987939+0000 mgr.smithi047.fuhzzr (mgr.14162) 49 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:50:08.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:08 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:50:07.148756+0000 mon.smithi047 (mon.0) 142 : audit [DBG] from='client.? 172.21.15.194:0/226719797' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:50:08.554 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:50:08.555 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:50:09.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:09 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:50:07.988306+0000 mgr.smithi047.fuhzzr (mgr.14162) 50 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:50:09.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:09.750+0000 7f7ca4d49700 1 -- 172.21.15.194:0/126835038 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ca00f6510 msgr2=0x7f7ca00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:09.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:09.750+0000 7f7ca4d49700 1 --2- 172.21.15.194:0/126835038 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ca00f6510 0x7f7ca00f6930 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f7c98009960 tx=0x7f7c98004d10).stop 2024-01-18T06:50:09.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:09.750+0000 7f7ca4d49700 1 -- 172.21.15.194:0/126835038 shutdown_connections 2024-01-18T06:50:09.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:09.750+0000 7f7ca4d49700 1 --2- 172.21.15.194:0/126835038 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ca00f6510 0x7f7ca00f6930 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:09.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:09.750+0000 7f7ca4d49700 1 -- 172.21.15.194:0/126835038 >> 172.21.15.194:0/126835038 conn(0x7f7ca00f1c70 msgr2=0x7f7ca00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:50:09.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:09.750+0000 7f7ca4d49700 1 -- 172.21.15.194:0/126835038 shutdown_connections 2024-01-18T06:50:09.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:09.750+0000 7f7ca4d49700 1 -- 172.21.15.194:0/126835038 wait complete. 2024-01-18T06:50:09.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:09.751+0000 7f7ca4d49700 1 Processor -- start 2024-01-18T06:50:09.756 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:09.751+0000 7f7ca4d49700 1 -- start start 2024-01-18T06:50:09.756 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:09.751+0000 7f7ca4d49700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ca00f6510 0x7f7ca0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:50:09.756 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:09.751+0000 7f7ca4d49700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca0106c20 con 0x7f7ca00f6510 2024-01-18T06:50:09.756 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:09.751+0000 7f7c9e59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ca00f6510 0x7f7ca0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:50:09.756 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:09.751+0000 7f7c9e59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ca00f6510 0x7f7ca0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:49670/0 (socket says 172.21.15.194:49670) 2024-01-18T06:50:09.756 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:09.751+0000 7f7c9e59c700 1 -- 172.21.15.194:0/424248424 learned_addr learned my addr 172.21.15.194:0/424248424 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:50:09.757 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:09.752+0000 7f7c9e59c700 1 -- 172.21.15.194:0/424248424 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c98009640 con 0x7f7ca00f6510 2024-01-18T06:50:09.757 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:09.752+0000 7f7c9e59c700 1 --2- 172.21.15.194:0/424248424 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ca00f6510 0x7f7ca0106730 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f7c98004b70 tx=0x7f7c98004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:50:09.757 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:09.752+0000 7f7c8effd700 1 -- 172.21.15.194:0/424248424 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c98018070 con 0x7f7ca00f6510 2024-01-18T06:50:09.757 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:09.752+0000 7f7ca4d49700 1 -- 172.21.15.194:0/424248424 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ca0106e20 con 0x7f7ca00f6510 2024-01-18T06:50:09.757 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:09.752+0000 7f7c8effd700 1 -- 172.21.15.194:0/424248424 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7c9800b870 con 0x7f7ca00f6510 2024-01-18T06:50:09.757 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:09.752+0000 7f7c8effd700 1 -- 172.21.15.194:0/424248424 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c9801c560 con 0x7f7ca00f6510 2024-01-18T06:50:09.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:09.752+0000 7f7ca4d49700 1 -- 172.21.15.194:0/424248424 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ca01072b0 con 0x7f7ca00f6510 2024-01-18T06:50:09.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:09.753+0000 7f7c8effd700 1 -- 172.21.15.194:0/424248424 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7c98013630 con 0x7f7ca00f6510 2024-01-18T06:50:09.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:09.753+0000 7f7ca4d49700 1 -- 172.21.15.194:0/424248424 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ca0045ad0 con 0x7f7ca00f6510 2024-01-18T06:50:09.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:09.753+0000 7f7c8effd700 1 --2- 172.21.15.194:0/424248424 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f7c8802ff60 0x7f7c88032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:50:09.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:09.753+0000 7f7c8effd700 1 -- 172.21.15.194:0/424248424 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7c98041290 con 0x7f7ca00f6510 2024-01-18T06:50:09.759 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:09.753+0000 7f7c9dd9b700 1 --2- 172.21.15.194:0/424248424 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f7c8802ff60 0x7f7c88032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:50:09.759 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:09.754+0000 7f7c9dd9b700 1 --2- 172.21.15.194:0/424248424 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f7c8802ff60 0x7f7c88032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f7c940096a0 tx=0x7f7c94006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:50:09.759 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:09.758+0000 7f7c8effd700 1 -- 172.21.15.194:0/424248424 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7c98004810 con 0x7f7ca00f6510 2024-01-18T06:50:09.963 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:09.963+0000 7f7ca4d49700 1 -- 172.21.15.194:0/424248424 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7ca00fbf60 con 0x7f7ca00f6510 2024-01-18T06:50:09.963 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:09.963+0000 7f7c8effd700 1 -- 172.21.15.194:0/424248424 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7c98004e10 con 0x7f7ca00f6510 2024-01-18T06:50:09.964 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:50:09.964 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:50:09.966 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:09.965+0000 7f7ca4d49700 1 -- 172.21.15.194:0/424248424 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f7c8802ff60 msgr2=0x7f7c88032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:09.966 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:09.965+0000 7f7ca4d49700 1 --2- 172.21.15.194:0/424248424 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f7c8802ff60 0x7f7c88032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f7c940096a0 tx=0x7f7c94006b40).stop 2024-01-18T06:50:09.966 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:09.965+0000 7f7ca4d49700 1 -- 172.21.15.194:0/424248424 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ca00f6510 msgr2=0x7f7ca0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:09.966 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:09.965+0000 7f7ca4d49700 1 --2- 172.21.15.194:0/424248424 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ca00f6510 0x7f7ca0106730 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f7c98004b70 tx=0x7f7c98004d10).stop 2024-01-18T06:50:09.966 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:09.965+0000 7f7ca4d49700 1 -- 172.21.15.194:0/424248424 shutdown_connections 2024-01-18T06:50:09.967 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:09.965+0000 7f7ca4d49700 1 --2- 172.21.15.194:0/424248424 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f7c8802ff60 0x7f7c88032420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:09.967 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:09.965+0000 7f7ca4d49700 1 --2- 172.21.15.194:0/424248424 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ca00f6510 0x7f7ca0106730 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:09.967 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:09.965+0000 7f7ca4d49700 1 -- 172.21.15.194:0/424248424 >> 172.21.15.194:0/424248424 conn(0x7f7ca00f1c70 msgr2=0x7f7ca00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:50:09.967 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:09.965+0000 7f7ca4d49700 1 -- 172.21.15.194:0/424248424 shutdown_connections 2024-01-18T06:50:09.967 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:09.965+0000 7f7ca4d49700 1 -- 172.21.15.194:0/424248424 wait complete. 2024-01-18T06:50:09.967 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:50:10.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:10 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:50:09.964431+0000 mon.smithi047 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.194:0/424248424' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:50:11.363 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:50:11.364 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:50:11.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:11 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:50:09.988671+0000 mgr.smithi047.fuhzzr (mgr.14162) 51 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:50:12.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:12.538+0000 7f5d03e2a700 1 -- 172.21.15.194:0/234965756 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5cfc060220 msgr2=0x7f5cfc0f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:12.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:12.538+0000 7f5d03e2a700 1 --2- 172.21.15.194:0/234965756 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5cfc060220 0x7f5cfc0f9150 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f5cf8009fb0 tx=0x7f5cf8009b30).stop 2024-01-18T06:50:12.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:12.538+0000 7f5d03e2a700 1 -- 172.21.15.194:0/234965756 shutdown_connections 2024-01-18T06:50:12.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:12.538+0000 7f5d03e2a700 1 --2- 172.21.15.194:0/234965756 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5cfc060220 0x7f5cfc0f9150 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:12.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:12.538+0000 7f5d03e2a700 1 -- 172.21.15.194:0/234965756 >> 172.21.15.194:0/234965756 conn(0x7f5cfc0f0b70 msgr2=0x7f5cfc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:50:12.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:12.538+0000 7f5d03e2a700 1 -- 172.21.15.194:0/234965756 shutdown_connections 2024-01-18T06:50:12.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:12.538+0000 7f5d03e2a700 1 -- 172.21.15.194:0/234965756 wait complete. 2024-01-18T06:50:12.539 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:12.538+0000 7f5d03e2a700 1 Processor -- start 2024-01-18T06:50:12.539 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:12.538+0000 7f5d03e2a700 1 -- start start 2024-01-18T06:50:12.539 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:12.538+0000 7f5d03e2a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5cfc060220 0x7f5cfc1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:50:12.539 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:12.538+0000 7f5d03e2a700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cfc106bd0 con 0x7f5cfc060220 2024-01-18T06:50:12.539 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:12.539+0000 7f5d01bc6700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5cfc060220 0x7f5cfc1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:50:12.539 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:12.539+0000 7f5d01bc6700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5cfc060220 0x7f5cfc1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:49680/0 (socket says 172.21.15.194:49680) 2024-01-18T06:50:12.539 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:12.539+0000 7f5d01bc6700 1 -- 172.21.15.194:0/822291785 learned_addr learned my addr 172.21.15.194:0/822291785 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:50:12.540 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:12.539+0000 7f5d01bc6700 1 -- 172.21.15.194:0/822291785 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf8005040 con 0x7f5cfc060220 2024-01-18T06:50:12.541 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:12.539+0000 7f5d01bc6700 1 --2- 172.21.15.194:0/822291785 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5cfc060220 0x7f5cfc1066e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f5cf8013510 tx=0x7f5cf800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:50:12.541 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:12.539+0000 7f5cee7fc700 1 -- 172.21.15.194:0/822291785 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cf8009320 con 0x7f5cfc060220 2024-01-18T06:50:12.541 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:12.539+0000 7f5cee7fc700 1 -- 172.21.15.194:0/822291785 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5cf8013f00 con 0x7f5cfc060220 2024-01-18T06:50:12.541 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:12.539+0000 7f5cee7fc700 1 -- 172.21.15.194:0/822291785 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cf8014dc0 con 0x7f5cfc060220 2024-01-18T06:50:12.541 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:12.539+0000 7f5d03e2a700 1 -- 172.21.15.194:0/822291785 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cfc106dd0 con 0x7f5cfc060220 2024-01-18T06:50:12.541 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:12.539+0000 7f5d03e2a700 1 -- 172.21.15.194:0/822291785 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cfc107260 con 0x7f5cfc060220 2024-01-18T06:50:12.541 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:12.540+0000 7f5cee7fc700 1 -- 172.21.15.194:0/822291785 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5cf80095f0 con 0x7f5cfc060220 2024-01-18T06:50:12.541 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:12.540+0000 7f5d03e2a700 1 -- 172.21.15.194:0/822291785 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cfc045ad0 con 0x7f5cfc060220 2024-01-18T06:50:12.541 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:12.540+0000 7f5cee7fc700 1 --2- 172.21.15.194:0/822291785 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f5ce802ff60 0x7f5ce8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:50:12.541 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:12.540+0000 7f5cee7fc700 1 -- 172.21.15.194:0/822291785 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5cf8041040 con 0x7f5cfc060220 2024-01-18T06:50:12.541 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:12.540+0000 7f5d013c5700 1 --2- 172.21.15.194:0/822291785 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f5ce802ff60 0x7f5ce8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:50:12.541 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:12.541+0000 7f5d013c5700 1 --2- 172.21.15.194:0/822291785 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f5ce802ff60 0x7f5ce8032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f5cf00096a0 tx=0x7f5cf0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:50:12.544 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:12.545+0000 7f5cee7fc700 1 -- 172.21.15.194:0/822291785 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5cf8014420 con 0x7f5cfc060220 2024-01-18T06:50:12.750 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:12.751+0000 7f5d03e2a700 1 -- 172.21.15.194:0/822291785 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5cfc108110 con 0x7f5cfc060220 2024-01-18T06:50:12.751 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:12.752+0000 7f5cee7fc700 1 -- 172.21.15.194:0/822291785 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5cf8010230 con 0x7f5cfc060220 2024-01-18T06:50:12.751 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:50:12.751 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:50:12.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:12.753+0000 7f5d03e2a700 1 -- 172.21.15.194:0/822291785 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f5ce802ff60 msgr2=0x7f5ce8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:12.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:12.753+0000 7f5d03e2a700 1 --2- 172.21.15.194:0/822291785 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f5ce802ff60 0x7f5ce8032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f5cf00096a0 tx=0x7f5cf0006b40).stop 2024-01-18T06:50:12.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:12.753+0000 7f5d03e2a700 1 -- 172.21.15.194:0/822291785 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5cfc060220 msgr2=0x7f5cfc1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:12.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:12.753+0000 7f5d03e2a700 1 --2- 172.21.15.194:0/822291785 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5cfc060220 0x7f5cfc1066e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f5cf8013510 tx=0x7f5cf800b040).stop 2024-01-18T06:50:12.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:12.753+0000 7f5d03e2a700 1 -- 172.21.15.194:0/822291785 shutdown_connections 2024-01-18T06:50:12.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:12.753+0000 7f5d03e2a700 1 --2- 172.21.15.194:0/822291785 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f5ce802ff60 0x7f5ce8032420 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:12.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:12.753+0000 7f5d03e2a700 1 --2- 172.21.15.194:0/822291785 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5cfc060220 0x7f5cfc1066e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:12.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:12.753+0000 7f5d03e2a700 1 -- 172.21.15.194:0/822291785 >> 172.21.15.194:0/822291785 conn(0x7f5cfc0f0b70 msgr2=0x7f5cfc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:50:12.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:12.753+0000 7f5d03e2a700 1 -- 172.21.15.194:0/822291785 shutdown_connections 2024-01-18T06:50:12.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:12.753+0000 7f5d03e2a700 1 -- 172.21.15.194:0/822291785 wait complete. 2024-01-18T06:50:12.754 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:50:13.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:13 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:50:11.989040+0000 mgr.smithi047.fuhzzr (mgr.14162) 52 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:50:13.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:13 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:50:12.752450+0000 mon.smithi047 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.194:0/822291785' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:50:14.137 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:50:14.138 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:50:15.464 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:15 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:50:13.989445+0000 mgr.smithi047.fuhzzr (mgr.14162) 53 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:50:15.465 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:15.465+0000 7fa7392d2700 1 -- 172.21.15.194:0/2555781145 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7340f6510 msgr2=0x7fa7340f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:15.465 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:15.465+0000 7fa7392d2700 1 --2- 172.21.15.194:0/2555781145 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7340f6510 0x7fa7340f6930 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fa728009fb0 tx=0x7fa728009b30).stop 2024-01-18T06:50:15.465 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:15.465+0000 7fa7392d2700 1 -- 172.21.15.194:0/2555781145 shutdown_connections 2024-01-18T06:50:15.465 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:15.465+0000 7fa7392d2700 1 --2- 172.21.15.194:0/2555781145 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7340f6510 0x7fa7340f6930 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:15.465 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:15.465+0000 7fa7392d2700 1 -- 172.21.15.194:0/2555781145 >> 172.21.15.194:0/2555781145 conn(0x7fa7340f1c70 msgr2=0x7fa7340f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:50:15.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:15.465+0000 7fa7392d2700 1 -- 172.21.15.194:0/2555781145 shutdown_connections 2024-01-18T06:50:15.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:15.465+0000 7fa7392d2700 1 -- 172.21.15.194:0/2555781145 wait complete. 2024-01-18T06:50:15.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:15.466+0000 7fa7392d2700 1 Processor -- start 2024-01-18T06:50:15.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:15.466+0000 7fa7392d2700 1 -- start start 2024-01-18T06:50:15.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:15.466+0000 7fa7392d2700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7340f6510 0x7fa734106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:50:15.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:15.466+0000 7fa7392d2700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa734106c20 con 0x7fa7340f6510 2024-01-18T06:50:15.467 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:15.466+0000 7fa732ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7340f6510 0x7fa734106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:50:15.468 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:15.466+0000 7fa732ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7340f6510 0x7fa734106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:49698/0 (socket says 172.21.15.194:49698) 2024-01-18T06:50:15.468 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:15.466+0000 7fa732ffd700 1 -- 172.21.15.194:0/2182928491 learned_addr learned my addr 172.21.15.194:0/2182928491 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:50:15.468 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:15.467+0000 7fa732ffd700 1 -- 172.21.15.194:0/2182928491 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa728005040 con 0x7fa7340f6510 2024-01-18T06:50:15.468 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:15.467+0000 7fa732ffd700 1 --2- 172.21.15.194:0/2182928491 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7340f6510 0x7fa734106730 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fa728013510 tx=0x7fa72800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:50:15.468 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:15.467+0000 7fa71b7fe700 1 -- 172.21.15.194:0/2182928491 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa728009320 con 0x7fa7340f6510 2024-01-18T06:50:15.468 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:15.467+0000 7fa7392d2700 1 -- 172.21.15.194:0/2182928491 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa734106e20 con 0x7fa7340f6510 2024-01-18T06:50:15.468 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:15.467+0000 7fa71b7fe700 1 -- 172.21.15.194:0/2182928491 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa728013f00 con 0x7fa7340f6510 2024-01-18T06:50:15.468 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:15.467+0000 7fa71b7fe700 1 -- 172.21.15.194:0/2182928491 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa728014d90 con 0x7fa7340f6510 2024-01-18T06:50:15.468 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:15.467+0000 7fa7392d2700 1 -- 172.21.15.194:0/2182928491 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7341072b0 con 0x7fa7340f6510 2024-01-18T06:50:15.468 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:15.468+0000 7fa71b7fe700 1 -- 172.21.15.194:0/2182928491 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa7280095f0 con 0x7fa7340f6510 2024-01-18T06:50:15.468 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:15.468+0000 7fa71b7fe700 1 --2- 172.21.15.194:0/2182928491 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa71c02ff60 0x7fa71c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:50:15.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:15.468+0000 7fa7392d2700 1 -- 172.21.15.194:0/2182928491 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa734045ad0 con 0x7fa7340f6510 2024-01-18T06:50:15.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:15.468+0000 7fa71b7fe700 1 -- 172.21.15.194:0/2182928491 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa728041080 con 0x7fa7340f6510 2024-01-18T06:50:15.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:15.468+0000 7fa7327fc700 1 --2- 172.21.15.194:0/2182928491 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa71c02ff60 0x7fa71c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:50:15.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:15.469+0000 7fa7327fc700 1 --2- 172.21.15.194:0/2182928491 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa71c02ff60 0x7fa71c032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fa7240096a0 tx=0x7fa724006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:50:15.472 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:15.473+0000 7fa71b7fe700 1 -- 172.21.15.194:0/2182928491 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa728014420 con 0x7fa7340f6510 2024-01-18T06:50:15.676 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:15.677+0000 7fa7392d2700 1 -- 172.21.15.194:0/2182928491 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa7340fc050 con 0x7fa7340f6510 2024-01-18T06:50:15.677 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:15.677+0000 7fa71b7fe700 1 -- 172.21.15.194:0/2182928491 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa728010230 con 0x7fa7340f6510 2024-01-18T06:50:15.677 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:50:15.677 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:50:15.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:15.679+0000 7fa7392d2700 1 -- 172.21.15.194:0/2182928491 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa71c02ff60 msgr2=0x7fa71c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:15.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:15.679+0000 7fa7392d2700 1 --2- 172.21.15.194:0/2182928491 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa71c02ff60 0x7fa71c032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fa7240096a0 tx=0x7fa724006b40).stop 2024-01-18T06:50:15.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:15.679+0000 7fa7392d2700 1 -- 172.21.15.194:0/2182928491 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7340f6510 msgr2=0x7fa734106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:15.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:15.679+0000 7fa7392d2700 1 --2- 172.21.15.194:0/2182928491 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7340f6510 0x7fa734106730 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fa728013510 tx=0x7fa72800b040).stop 2024-01-18T06:50:15.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:15.679+0000 7fa7392d2700 1 -- 172.21.15.194:0/2182928491 shutdown_connections 2024-01-18T06:50:15.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:15.679+0000 7fa7392d2700 1 --2- 172.21.15.194:0/2182928491 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa71c02ff60 0x7fa71c032420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:15.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:15.679+0000 7fa7392d2700 1 --2- 172.21.15.194:0/2182928491 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7340f6510 0x7fa734106730 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:15.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:15.679+0000 7fa7392d2700 1 -- 172.21.15.194:0/2182928491 >> 172.21.15.194:0/2182928491 conn(0x7fa7340f1c70 msgr2=0x7fa7340f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:50:15.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:15.679+0000 7fa7392d2700 1 -- 172.21.15.194:0/2182928491 shutdown_connections 2024-01-18T06:50:15.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:15.679+0000 7fa7392d2700 1 -- 172.21.15.194:0/2182928491 wait complete. 2024-01-18T06:50:15.680 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:50:16.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:16 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:50:15.678414+0000 mon.smithi047 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.194:0/2182928491' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:50:17.016 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:50:17.017 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:50:17.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:17 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:50:15.989870+0000 mgr.smithi047.fuhzzr (mgr.14162) 54 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:50:18.225 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:18.225+0000 7fb6e32ae700 1 -- 172.21.15.194:0/1671910279 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6dc0f51a0 msgr2=0x7fb6dc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:18.226 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:18.225+0000 7fb6e32ae700 1 --2- 172.21.15.194:0/1671910279 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6dc0f51a0 0x7fb6dc0f55c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fb6cc009960 tx=0x7fb6cc004d10).stop 2024-01-18T06:50:18.226 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:18.225+0000 7fb6e32ae700 1 -- 172.21.15.194:0/1671910279 shutdown_connections 2024-01-18T06:50:18.226 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:18.225+0000 7fb6e32ae700 1 --2- 172.21.15.194:0/1671910279 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6dc0f51a0 0x7fb6dc0f55c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:18.226 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:18.225+0000 7fb6e32ae700 1 -- 172.21.15.194:0/1671910279 >> 172.21.15.194:0/1671910279 conn(0x7fb6dc0f09a0 msgr2=0x7fb6dc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:50:18.226 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:18.225+0000 7fb6e32ae700 1 -- 172.21.15.194:0/1671910279 shutdown_connections 2024-01-18T06:50:18.226 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:18.225+0000 7fb6e32ae700 1 -- 172.21.15.194:0/1671910279 wait complete. 2024-01-18T06:50:18.226 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:18.225+0000 7fb6e32ae700 1 Processor -- start 2024-01-18T06:50:18.226 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:18.226+0000 7fb6e32ae700 1 -- start start 2024-01-18T06:50:18.226 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:18.226+0000 7fb6e32ae700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6dc0f51a0 0x7fb6dc1053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:50:18.226 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:18.226+0000 7fb6e32ae700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6dc1058a0 con 0x7fb6dc0f51a0 2024-01-18T06:50:18.226 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:18.226+0000 7fb6e22ac700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6dc0f51a0 0x7fb6dc1053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:50:18.227 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:18.226+0000 7fb6e22ac700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6dc0f51a0 0x7fb6dc1053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:35850/0 (socket says 172.21.15.194:35850) 2024-01-18T06:50:18.227 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:18.226+0000 7fb6e22ac700 1 -- 172.21.15.194:0/1749057619 learned_addr learned my addr 172.21.15.194:0/1749057619 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:50:18.228 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:18.226+0000 7fb6e22ac700 1 -- 172.21.15.194:0/1749057619 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6cc009640 con 0x7fb6dc0f51a0 2024-01-18T06:50:18.228 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:18.227+0000 7fb6e22ac700 1 --2- 172.21.15.194:0/1749057619 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6dc0f51a0 0x7fb6dc1053b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fb6cc007bd0 tx=0x7fb6cc004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:50:18.228 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:18.227+0000 7fb6daffd700 1 -- 172.21.15.194:0/1749057619 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6cc018070 con 0x7fb6dc0f51a0 2024-01-18T06:50:18.229 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:18.227+0000 7fb6e32ae700 1 -- 172.21.15.194:0/1749057619 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6dc105aa0 con 0x7fb6dc0f51a0 2024-01-18T06:50:18.229 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:18.227+0000 7fb6daffd700 1 -- 172.21.15.194:0/1749057619 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb6cc00b870 con 0x7fb6dc0f51a0 2024-01-18T06:50:18.229 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:18.227+0000 7fb6daffd700 1 -- 172.21.15.194:0/1749057619 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6cc01d420 con 0x7fb6dc0f51a0 2024-01-18T06:50:18.229 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:18.227+0000 7fb6e32ae700 1 -- 172.21.15.194:0/1749057619 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6dc105f30 con 0x7fb6dc0f51a0 2024-01-18T06:50:18.229 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:18.228+0000 7fb6daffd700 1 -- 172.21.15.194:0/1749057619 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb6cc024060 con 0x7fb6dc0f51a0 2024-01-18T06:50:18.229 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:18.228+0000 7fb6e32ae700 1 -- 172.21.15.194:0/1749057619 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6dc045ad0 con 0x7fb6dc0f51a0 2024-01-18T06:50:18.229 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:18.228+0000 7fb6daffd700 1 --2- 172.21.15.194:0/1749057619 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb6c402ff60 0x7fb6c4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:50:18.229 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:18.228+0000 7fb6daffd700 1 -- 172.21.15.194:0/1749057619 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb6cc041170 con 0x7fb6dc0f51a0 2024-01-18T06:50:18.229 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:18.228+0000 7fb6e1aab700 1 --2- 172.21.15.194:0/1749057619 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb6c402ff60 0x7fb6c4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:50:18.229 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:18.228+0000 7fb6e1aab700 1 --2- 172.21.15.194:0/1749057619 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb6c402ff60 0x7fb6c4032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fb6d00096a0 tx=0x7fb6d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:50:18.232 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:18.233+0000 7fb6daffd700 1 -- 172.21.15.194:0/1749057619 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb6cc025520 con 0x7fb6dc0f51a0 2024-01-18T06:50:18.436 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:18.437+0000 7fb6e32ae700 1 -- 172.21.15.194:0/1749057619 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb6dc106d40 con 0x7fb6dc0f51a0 2024-01-18T06:50:18.437 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:18.438+0000 7fb6daffd700 1 -- 172.21.15.194:0/1749057619 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb6cc01a2d0 con 0x7fb6dc0f51a0 2024-01-18T06:50:18.437 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:50:18.437 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:50:18.439 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:18.439+0000 7fb6e32ae700 1 -- 172.21.15.194:0/1749057619 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb6c402ff60 msgr2=0x7fb6c4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:18.439 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:18.439+0000 7fb6e32ae700 1 --2- 172.21.15.194:0/1749057619 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb6c402ff60 0x7fb6c4032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fb6d00096a0 tx=0x7fb6d0006b40).stop 2024-01-18T06:50:18.439 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:18.439+0000 7fb6e32ae700 1 -- 172.21.15.194:0/1749057619 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6dc0f51a0 msgr2=0x7fb6dc1053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:18.439 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:18.439+0000 7fb6e32ae700 1 --2- 172.21.15.194:0/1749057619 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6dc0f51a0 0x7fb6dc1053b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fb6cc007bd0 tx=0x7fb6cc004d10).stop 2024-01-18T06:50:18.439 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:18.439+0000 7fb6e32ae700 1 -- 172.21.15.194:0/1749057619 shutdown_connections 2024-01-18T06:50:18.439 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:18.439+0000 7fb6e32ae700 1 --2- 172.21.15.194:0/1749057619 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb6c402ff60 0x7fb6c4032420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:18.439 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:18.439+0000 7fb6e32ae700 1 --2- 172.21.15.194:0/1749057619 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb6dc0f51a0 0x7fb6dc1053b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:18.439 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:18.439+0000 7fb6e32ae700 1 -- 172.21.15.194:0/1749057619 >> 172.21.15.194:0/1749057619 conn(0x7fb6dc0f09a0 msgr2=0x7fb6dc0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:50:18.439 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:18.439+0000 7fb6e32ae700 1 -- 172.21.15.194:0/1749057619 shutdown_connections 2024-01-18T06:50:18.439 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:18.439+0000 7fb6e32ae700 1 -- 172.21.15.194:0/1749057619 wait complete. 2024-01-18T06:50:18.440 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:50:19.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:18 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:50:17.990249+0000 mgr.smithi047.fuhzzr (mgr.14162) 55 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:50:19.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:18 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:50:18.438423+0000 mon.smithi047 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.194:0/1749057619' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:50:19.814 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:50:19.814 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:50:21.005 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:21.005+0000 7f405ef0c700 1 -- 172.21.15.194:0/288909898 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40580f5410 msgr2=0x7f40580f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:21.005 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:21.005+0000 7f405ef0c700 1 --2- 172.21.15.194:0/288909898 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40580f5410 0x7f40580f5830 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f404c009fb0 tx=0x7f404c009b30).stop 2024-01-18T06:50:21.006 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:21.005+0000 7f405ef0c700 1 -- 172.21.15.194:0/288909898 shutdown_connections 2024-01-18T06:50:21.006 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:21.005+0000 7f405ef0c700 1 --2- 172.21.15.194:0/288909898 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40580f5410 0x7f40580f5830 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:21.006 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:21.005+0000 7f405ef0c700 1 -- 172.21.15.194:0/288909898 >> 172.21.15.194:0/288909898 conn(0x7f40580f0b70 msgr2=0x7f40580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:50:21.006 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:21.005+0000 7f405ef0c700 1 -- 172.21.15.194:0/288909898 shutdown_connections 2024-01-18T06:50:21.006 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:21.005+0000 7f405ef0c700 1 -- 172.21.15.194:0/288909898 wait complete. 2024-01-18T06:50:21.006 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:21.006+0000 7f405ef0c700 1 Processor -- start 2024-01-18T06:50:21.006 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:21.006+0000 7f405ef0c700 1 -- start start 2024-01-18T06:50:21.006 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:21.006+0000 7f405ef0c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40580f5410 0x7f40581055d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:50:21.006 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:21.006+0000 7f405ef0c700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4058105ac0 con 0x7f40580f5410 2024-01-18T06:50:21.007 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:21.006+0000 7f405cca8700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40580f5410 0x7f40581055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:50:21.008 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:21.006+0000 7f405cca8700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40580f5410 0x7f40581055d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:35870/0 (socket says 172.21.15.194:35870) 2024-01-18T06:50:21.008 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:21.006+0000 7f405cca8700 1 -- 172.21.15.194:0/2784874450 learned_addr learned my addr 172.21.15.194:0/2784874450 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:50:21.008 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:21.007+0000 7f405cca8700 1 -- 172.21.15.194:0/2784874450 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f404c005040 con 0x7f40580f5410 2024-01-18T06:50:21.008 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:21.007+0000 7f405cca8700 1 --2- 172.21.15.194:0/2784874450 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40580f5410 0x7f40581055d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f404c014000 tx=0x7f404c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:50:21.008 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:21.007+0000 7f40557fa700 1 -- 172.21.15.194:0/2784874450 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f404c0095f0 con 0x7f40580f5410 2024-01-18T06:50:21.008 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:21.007+0000 7f405ef0c700 1 -- 172.21.15.194:0/2784874450 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4058105cc0 con 0x7f40580f5410 2024-01-18T06:50:21.008 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:21.007+0000 7f40557fa700 1 -- 172.21.15.194:0/2784874450 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f404c007e70 con 0x7f40580f5410 2024-01-18T06:50:21.008 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:21.007+0000 7f40557fa700 1 -- 172.21.15.194:0/2784874450 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f404c014ea0 con 0x7f40580f5410 2024-01-18T06:50:21.008 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:21.007+0000 7f405ef0c700 1 -- 172.21.15.194:0/2784874450 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4058106150 con 0x7f40580f5410 2024-01-18T06:50:21.008 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:21.008+0000 7f40557fa700 1 -- 172.21.15.194:0/2784874450 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f404c025070 con 0x7f40580f5410 2024-01-18T06:50:21.008 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:21.008+0000 7f405ef0c700 1 -- 172.21.15.194:0/2784874450 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4058045ad0 con 0x7f40580f5410 2024-01-18T06:50:21.008 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:21.008+0000 7f40557fa700 1 --2- 172.21.15.194:0/2784874450 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f404802ff60 0x7f4048032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:50:21.009 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:21.008+0000 7f40557fa700 1 -- 172.21.15.194:0/2784874450 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f404c040940 con 0x7f40580f5410 2024-01-18T06:50:21.009 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:21.008+0000 7f4057fff700 1 --2- 172.21.15.194:0/2784874450 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f404802ff60 0x7f4048032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:50:21.009 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:21.009+0000 7f4057fff700 1 --2- 172.21.15.194:0/2784874450 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f404802ff60 0x7f4048032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f40440096a0 tx=0x7f4044006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:50:21.012 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:21.013+0000 7f40557fa700 1 -- 172.21.15.194:0/2784874450 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f404c0079c0 con 0x7f40580f5410 2024-01-18T06:50:21.219 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:21.220+0000 7f405ef0c700 1 -- 172.21.15.194:0/2784874450 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4058106fa0 con 0x7f40580f5410 2024-01-18T06:50:21.220 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:21.221+0000 7f40557fa700 1 -- 172.21.15.194:0/2784874450 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f404c00f300 con 0x7f40580f5410 2024-01-18T06:50:21.220 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:50:21.220 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:50:21.222 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:21.222+0000 7f405ef0c700 1 -- 172.21.15.194:0/2784874450 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f404802ff60 msgr2=0x7f4048032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:21.222 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:21.222+0000 7f405ef0c700 1 --2- 172.21.15.194:0/2784874450 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f404802ff60 0x7f4048032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f40440096a0 tx=0x7f4044006b40).stop 2024-01-18T06:50:21.222 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:21.222+0000 7f405ef0c700 1 -- 172.21.15.194:0/2784874450 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40580f5410 msgr2=0x7f40581055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:21.222 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:21.222+0000 7f405ef0c700 1 --2- 172.21.15.194:0/2784874450 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40580f5410 0x7f40581055d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f404c014000 tx=0x7f404c00b040).stop 2024-01-18T06:50:21.222 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:21.222+0000 7f405ef0c700 1 -- 172.21.15.194:0/2784874450 shutdown_connections 2024-01-18T06:50:21.222 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:21.222+0000 7f405ef0c700 1 --2- 172.21.15.194:0/2784874450 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f404802ff60 0x7f4048032420 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:21.222 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:21.222+0000 7f405ef0c700 1 --2- 172.21.15.194:0/2784874450 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40580f5410 0x7f40581055d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:21.222 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:21.222+0000 7f405ef0c700 1 -- 172.21.15.194:0/2784874450 >> 172.21.15.194:0/2784874450 conn(0x7f40580f0b70 msgr2=0x7f40580f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:50:21.222 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:21.222+0000 7f405ef0c700 1 -- 172.21.15.194:0/2784874450 shutdown_connections 2024-01-18T06:50:21.222 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:21.222+0000 7f405ef0c700 1 -- 172.21.15.194:0/2784874450 wait complete. 2024-01-18T06:50:21.223 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:50:21.464 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:21 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:50:19.990550+0000 mgr.smithi047.fuhzzr (mgr.14162) 56 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:50:22.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:22 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:50:21.221594+0000 mon.smithi047 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.194:0/2784874450' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:50:22.605 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:50:22.605 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:50:23.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:23 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:50:21.990894+0000 mgr.smithi047.fuhzzr (mgr.14162) 57 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:50:23.829 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:23.829+0000 7f43bffff700 1 -- 172.21.15.194:0/2475041822 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43c00f4920 msgr2=0x7f43c00f4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:23.829 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:23.829+0000 7f43bffff700 1 --2- 172.21.15.194:0/2475041822 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43c00f4920 0x7f43c00f4d40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f43b0009fb0 tx=0x7f43b0009b30).stop 2024-01-18T06:50:23.829 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:23.829+0000 7f43bffff700 1 -- 172.21.15.194:0/2475041822 shutdown_connections 2024-01-18T06:50:23.829 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:23.829+0000 7f43bffff700 1 --2- 172.21.15.194:0/2475041822 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43c00f4920 0x7f43c00f4d40 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:23.829 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:23.829+0000 7f43bffff700 1 -- 172.21.15.194:0/2475041822 >> 172.21.15.194:0/2475041822 conn(0x7f43c00f0120 msgr2=0x7f43c00f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:50:23.829 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:23.829+0000 7f43bffff700 1 -- 172.21.15.194:0/2475041822 shutdown_connections 2024-01-18T06:50:23.829 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:23.829+0000 7f43bffff700 1 -- 172.21.15.194:0/2475041822 wait complete. 2024-01-18T06:50:23.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:23.830+0000 7f43bffff700 1 Processor -- start 2024-01-18T06:50:23.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:23.830+0000 7f43bffff700 1 -- start start 2024-01-18T06:50:23.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:23.830+0000 7f43bffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43c00f4920 0x7f43c0104b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:50:23.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:23.830+0000 7f43bffff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43c0105010 con 0x7f43c00f4920 2024-01-18T06:50:23.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:23.830+0000 7f43beffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43c00f4920 0x7f43c0104b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:50:23.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:23.830+0000 7f43beffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43c00f4920 0x7f43c0104b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:35882/0 (socket says 172.21.15.194:35882) 2024-01-18T06:50:23.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:23.830+0000 7f43beffd700 1 -- 172.21.15.194:0/625407836 learned_addr learned my addr 172.21.15.194:0/625407836 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:50:23.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:23.831+0000 7f43beffd700 1 -- 172.21.15.194:0/625407836 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b0005040 con 0x7f43c00f4920 2024-01-18T06:50:23.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:23.831+0000 7f43beffd700 1 --2- 172.21.15.194:0/625407836 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43c00f4920 0x7f43c0104b20 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f43b0013510 tx=0x7f43b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:50:23.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:23.831+0000 7f43a77fe700 1 -- 172.21.15.194:0/625407836 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43b0009320 con 0x7f43c00f4920 2024-01-18T06:50:23.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:23.831+0000 7f43bffff700 1 -- 172.21.15.194:0/625407836 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43c0105210 con 0x7f43c00f4920 2024-01-18T06:50:23.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:23.831+0000 7f43bffff700 1 -- 172.21.15.194:0/625407836 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43c01056a0 con 0x7f43c00f4920 2024-01-18T06:50:23.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:23.831+0000 7f43a77fe700 1 -- 172.21.15.194:0/625407836 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f43b0013f00 con 0x7f43c00f4920 2024-01-18T06:50:23.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:23.831+0000 7f43a77fe700 1 -- 172.21.15.194:0/625407836 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43b0014c10 con 0x7f43c00f4920 2024-01-18T06:50:23.833 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:23.832+0000 7f43bffff700 1 -- 172.21.15.194:0/625407836 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43a0004fa0 con 0x7f43c00f4920 2024-01-18T06:50:23.833 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:23.832+0000 7f43a77fe700 1 -- 172.21.15.194:0/625407836 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f43b0014d70 con 0x7f43c00f4920 2024-01-18T06:50:23.833 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:23.832+0000 7f43a77fe700 1 --2- 172.21.15.194:0/625407836 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f43a802ff60 0x7f43a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:50:23.833 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:23.832+0000 7f43a77fe700 1 -- 172.21.15.194:0/625407836 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f43b0041ec0 con 0x7f43c00f4920 2024-01-18T06:50:23.833 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:23.832+0000 7f43be7fc700 1 --2- 172.21.15.194:0/625407836 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f43a802ff60 0x7f43a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:50:23.833 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:23.833+0000 7f43be7fc700 1 --2- 172.21.15.194:0/625407836 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f43a802ff60 0x7f43a8032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f43b40096a0 tx=0x7f43b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:50:23.836 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:23.837+0000 7f43a77fe700 1 -- 172.21.15.194:0/625407836 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f43b0014420 con 0x7f43c00f4920 2024-01-18T06:50:24.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:24.039+0000 7f43bffff700 1 -- 172.21.15.194:0/625407836 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f43a0005670 con 0x7f43c00f4920 2024-01-18T06:50:24.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:24.040+0000 7f43a77fe700 1 -- 172.21.15.194:0/625407836 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f43b0024250 con 0x7f43c00f4920 2024-01-18T06:50:24.039 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:50:24.039 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:50:24.045 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:24.041+0000 7f43bffff700 1 -- 172.21.15.194:0/625407836 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f43a802ff60 msgr2=0x7f43a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:24.045 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:24.041+0000 7f43bffff700 1 --2- 172.21.15.194:0/625407836 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f43a802ff60 0x7f43a8032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f43b40096a0 tx=0x7f43b4006b40).stop 2024-01-18T06:50:24.045 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:24.041+0000 7f43bffff700 1 -- 172.21.15.194:0/625407836 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43c00f4920 msgr2=0x7f43c0104b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:24.045 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:24.041+0000 7f43bffff700 1 --2- 172.21.15.194:0/625407836 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43c00f4920 0x7f43c0104b20 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f43b0013510 tx=0x7f43b000b040).stop 2024-01-18T06:50:24.045 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:24.041+0000 7f43bffff700 1 -- 172.21.15.194:0/625407836 shutdown_connections 2024-01-18T06:50:24.045 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:24.041+0000 7f43bffff700 1 --2- 172.21.15.194:0/625407836 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f43a802ff60 0x7f43a8032420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:24.046 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:24.041+0000 7f43bffff700 1 --2- 172.21.15.194:0/625407836 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f43c00f4920 0x7f43c0104b20 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:24.046 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:24.041+0000 7f43bffff700 1 -- 172.21.15.194:0/625407836 >> 172.21.15.194:0/625407836 conn(0x7f43c00f0120 msgr2=0x7f43c00f0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:50:24.046 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:24.041+0000 7f43bffff700 1 -- 172.21.15.194:0/625407836 shutdown_connections 2024-01-18T06:50:24.046 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:24.041+0000 7f43bffff700 1 -- 172.21.15.194:0/625407836 wait complete. 2024-01-18T06:50:24.046 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:50:24.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:24 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:50:24.040593+0000 mon.smithi047 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.194:0/625407836' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:50:25.441 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:50:25.441 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:50:25.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:25 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:50:23.991249+0000 mgr.smithi047.fuhzzr (mgr.14162) 58 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:50:26.607 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:26.605+0000 7ff2fc747700 1 -- 172.21.15.194:0/3010876093 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2f40f5410 msgr2=0x7ff2f40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:26.608 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:26.605+0000 7ff2fc747700 1 --2- 172.21.15.194:0/3010876093 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2f40f5410 0x7ff2f40f5830 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7ff2e4009fb0 tx=0x7ff2e4009b30).stop 2024-01-18T06:50:26.608 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:26.605+0000 7ff2fc747700 1 -- 172.21.15.194:0/3010876093 shutdown_connections 2024-01-18T06:50:26.608 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:26.605+0000 7ff2fc747700 1 --2- 172.21.15.194:0/3010876093 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2f40f5410 0x7ff2f40f5830 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:26.608 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:26.605+0000 7ff2fc747700 1 -- 172.21.15.194:0/3010876093 >> 172.21.15.194:0/3010876093 conn(0x7ff2f40f0b70 msgr2=0x7ff2f40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:50:26.608 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:26.605+0000 7ff2fc747700 1 -- 172.21.15.194:0/3010876093 shutdown_connections 2024-01-18T06:50:26.608 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:26.605+0000 7ff2fc747700 1 -- 172.21.15.194:0/3010876093 wait complete. 2024-01-18T06:50:26.609 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:26.606+0000 7ff2fc747700 1 Processor -- start 2024-01-18T06:50:26.609 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:26.606+0000 7ff2fc747700 1 -- start start 2024-01-18T06:50:26.609 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:26.606+0000 7ff2fc747700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2f40f5410 0x7ff2f4106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:50:26.609 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:26.606+0000 7ff2fc747700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f4106bf0 con 0x7ff2f40f5410 2024-01-18T06:50:26.609 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:26.606+0000 7ff2fa4e3700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2f40f5410 0x7ff2f4106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:50:26.609 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:26.606+0000 7ff2fa4e3700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2f40f5410 0x7ff2f4106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:35896/0 (socket says 172.21.15.194:35896) 2024-01-18T06:50:26.609 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:26.606+0000 7ff2fa4e3700 1 -- 172.21.15.194:0/2305907521 learned_addr learned my addr 172.21.15.194:0/2305907521 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:50:26.609 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:26.606+0000 7ff2fa4e3700 1 -- 172.21.15.194:0/2305907521 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e4005040 con 0x7ff2f40f5410 2024-01-18T06:50:26.609 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:26.606+0000 7ff2fa4e3700 1 --2- 172.21.15.194:0/2305907521 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2f40f5410 0x7ff2f4106700 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7ff2e4013510 tx=0x7ff2e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:50:26.609 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:26.606+0000 7ff2eaffd700 1 -- 172.21.15.194:0/2305907521 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2e4009320 con 0x7ff2f40f5410 2024-01-18T06:50:26.609 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:26.607+0000 7ff2fc747700 1 -- 172.21.15.194:0/2305907521 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2f4106df0 con 0x7ff2f40f5410 2024-01-18T06:50:26.609 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:26.607+0000 7ff2eaffd700 1 -- 172.21.15.194:0/2305907521 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff2e4013f00 con 0x7ff2f40f5410 2024-01-18T06:50:26.609 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:26.607+0000 7ff2eaffd700 1 -- 172.21.15.194:0/2305907521 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2e4014d90 con 0x7ff2f40f5410 2024-01-18T06:50:26.609 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:26.607+0000 7ff2fc747700 1 -- 172.21.15.194:0/2305907521 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2f4107280 con 0x7ff2f40f5410 2024-01-18T06:50:26.610 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:26.607+0000 7ff2eaffd700 1 -- 172.21.15.194:0/2305907521 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff2e40095f0 con 0x7ff2f40f5410 2024-01-18T06:50:26.610 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:26.607+0000 7ff2fc747700 1 -- 172.21.15.194:0/2305907521 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2f4045ad0 con 0x7ff2f40f5410 2024-01-18T06:50:26.610 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:26.607+0000 7ff2eaffd700 1 --2- 172.21.15.194:0/2305907521 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7ff2e002ff60 0x7ff2e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:50:26.610 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:26.608+0000 7ff2eaffd700 1 -- 172.21.15.194:0/2305907521 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff2e4040fe0 con 0x7ff2f40f5410 2024-01-18T06:50:26.610 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:26.608+0000 7ff2f9ce2700 1 --2- 172.21.15.194:0/2305907521 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7ff2e002ff60 0x7ff2e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:50:26.610 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:26.608+0000 7ff2f9ce2700 1 --2- 172.21.15.194:0/2305907521 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7ff2e002ff60 0x7ff2e0032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7ff2f00096a0 tx=0x7ff2f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:50:26.611 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:26.612+0000 7ff2eaffd700 1 -- 172.21.15.194:0/2305907521 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff2e401d7f0 con 0x7ff2f40f5410 2024-01-18T06:50:26.816 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:26.817+0000 7ff2fc747700 1 -- 172.21.15.194:0/2305907521 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff2f40fae60 con 0x7ff2f40f5410 2024-01-18T06:50:26.817 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:26.817+0000 7ff2eaffd700 1 -- 172.21.15.194:0/2305907521 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff2e4010230 con 0x7ff2f40f5410 2024-01-18T06:50:26.817 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:50:26.817 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:50:26.819 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:26.819+0000 7ff2fc747700 1 -- 172.21.15.194:0/2305907521 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7ff2e002ff60 msgr2=0x7ff2e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:26.819 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:26.819+0000 7ff2fc747700 1 --2- 172.21.15.194:0/2305907521 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7ff2e002ff60 0x7ff2e0032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7ff2f00096a0 tx=0x7ff2f0006b40).stop 2024-01-18T06:50:26.819 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:26.819+0000 7ff2fc747700 1 -- 172.21.15.194:0/2305907521 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2f40f5410 msgr2=0x7ff2f4106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:26.819 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:26.819+0000 7ff2fc747700 1 --2- 172.21.15.194:0/2305907521 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2f40f5410 0x7ff2f4106700 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7ff2e4013510 tx=0x7ff2e400b040).stop 2024-01-18T06:50:26.819 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:26.819+0000 7ff2fc747700 1 -- 172.21.15.194:0/2305907521 shutdown_connections 2024-01-18T06:50:26.819 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:26.819+0000 7ff2fc747700 1 --2- 172.21.15.194:0/2305907521 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7ff2e002ff60 0x7ff2e0032420 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:26.820 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:26.819+0000 7ff2fc747700 1 --2- 172.21.15.194:0/2305907521 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2f40f5410 0x7ff2f4106700 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:26.820 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:26.819+0000 7ff2fc747700 1 -- 172.21.15.194:0/2305907521 >> 172.21.15.194:0/2305907521 conn(0x7ff2f40f0b70 msgr2=0x7ff2f40f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:50:26.820 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:26.819+0000 7ff2fc747700 1 -- 172.21.15.194:0/2305907521 shutdown_connections 2024-01-18T06:50:26.820 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:26.819+0000 7ff2fc747700 1 -- 172.21.15.194:0/2305907521 wait complete. 2024-01-18T06:50:26.820 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:50:27.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:27 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:50:25.991630+0000 mgr.smithi047.fuhzzr (mgr.14162) 59 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:50:27.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:27 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:50:26.818328+0000 mon.smithi047 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.194:0/2305907521' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:50:28.222 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:50:28.222 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:50:29.214 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:28 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:50:27.991964+0000 mgr.smithi047.fuhzzr (mgr.14162) 60 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:50:29.402 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:29.403+0000 7f10d759e700 1 -- 172.21.15.194:0/447606999 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f10d80f62a0 msgr2=0x7f10d80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:29.403 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:29.403+0000 7f10d759e700 1 --2- 172.21.15.194:0/447606999 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f10d80f62a0 0x7f10d80f66c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f10cc009fb0 tx=0x7f10cc009b30).stop 2024-01-18T06:50:29.403 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:29.403+0000 7f10d759e700 1 -- 172.21.15.194:0/447606999 shutdown_connections 2024-01-18T06:50:29.403 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:29.403+0000 7f10d759e700 1 --2- 172.21.15.194:0/447606999 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f10d80f62a0 0x7f10d80f66c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:29.403 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:29.403+0000 7f10d759e700 1 -- 172.21.15.194:0/447606999 >> 172.21.15.194:0/447606999 conn(0x7f10d80f1aa0 msgr2=0x7f10d80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:50:29.403 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:29.403+0000 7f10d759e700 1 -- 172.21.15.194:0/447606999 shutdown_connections 2024-01-18T06:50:29.403 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:29.403+0000 7f10d759e700 1 -- 172.21.15.194:0/447606999 wait complete. 2024-01-18T06:50:29.404 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:29.404+0000 7f10d759e700 1 Processor -- start 2024-01-18T06:50:29.404 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:29.404+0000 7f10d759e700 1 -- start start 2024-01-18T06:50:29.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:29.404+0000 7f10d759e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f10d80f62a0 0x7f10d81064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:50:29.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:29.404+0000 7f10d759e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10d81069a0 con 0x7f10d80f62a0 2024-01-18T06:50:29.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:29.404+0000 7f10d659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f10d80f62a0 0x7f10d81064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:50:29.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:29.404+0000 7f10d659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f10d80f62a0 0x7f10d81064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:36870/0 (socket says 172.21.15.194:36870) 2024-01-18T06:50:29.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:29.404+0000 7f10d659c700 1 -- 172.21.15.194:0/2496481921 learned_addr learned my addr 172.21.15.194:0/2496481921 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:50:29.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:29.405+0000 7f10d659c700 1 -- 172.21.15.194:0/2496481921 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10cc005040 con 0x7f10d80f62a0 2024-01-18T06:50:29.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:29.405+0000 7f10d659c700 1 --2- 172.21.15.194:0/2496481921 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f10d80f62a0 0x7f10d81064b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f10cc013510 tx=0x7f10cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:50:29.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:29.405+0000 7f10c6ffd700 1 -- 172.21.15.194:0/2496481921 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10cc009320 con 0x7f10d80f62a0 2024-01-18T06:50:29.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:29.405+0000 7f10d759e700 1 -- 172.21.15.194:0/2496481921 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10d8106ba0 con 0x7f10d80f62a0 2024-01-18T06:50:29.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:29.405+0000 7f10c6ffd700 1 -- 172.21.15.194:0/2496481921 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f10cc013f00 con 0x7f10d80f62a0 2024-01-18T06:50:29.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:29.405+0000 7f10c6ffd700 1 -- 172.21.15.194:0/2496481921 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10cc014dc0 con 0x7f10d80f62a0 2024-01-18T06:50:29.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:29.405+0000 7f10d759e700 1 -- 172.21.15.194:0/2496481921 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10d8107030 con 0x7f10d80f62a0 2024-01-18T06:50:29.406 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:29.406+0000 7f10c6ffd700 1 -- 172.21.15.194:0/2496481921 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f10cc0095f0 con 0x7f10d80f62a0 2024-01-18T06:50:29.406 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:29.406+0000 7f10d759e700 1 -- 172.21.15.194:0/2496481921 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10d8045ad0 con 0x7f10d80f62a0 2024-01-18T06:50:29.406 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:29.406+0000 7f10c6ffd700 1 --2- 172.21.15.194:0/2496481921 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f10c002ff60 0x7f10c0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:50:29.406 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:29.406+0000 7f10c6ffd700 1 -- 172.21.15.194:0/2496481921 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f10cc041000 con 0x7f10d80f62a0 2024-01-18T06:50:29.406 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:29.406+0000 7f10d5d9b700 1 --2- 172.21.15.194:0/2496481921 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f10c002ff60 0x7f10c0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:50:29.406 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:29.407+0000 7f10d5d9b700 1 --2- 172.21.15.194:0/2496481921 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f10c002ff60 0x7f10c0032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f10c80096a0 tx=0x7f10c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:50:29.410 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:29.411+0000 7f10c6ffd700 1 -- 172.21.15.194:0/2496481921 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f10cc01c790 con 0x7f10d80f62a0 2024-01-18T06:50:29.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:29.618+0000 7f10d759e700 1 -- 172.21.15.194:0/2496481921 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f10d8107ee0 con 0x7f10d80f62a0 2024-01-18T06:50:29.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:29.619+0000 7f10c6ffd700 1 -- 172.21.15.194:0/2496481921 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f10cc010230 con 0x7f10d80f62a0 2024-01-18T06:50:29.619 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:50:29.619 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:50:29.620 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:29.620+0000 7f10d759e700 1 -- 172.21.15.194:0/2496481921 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f10c002ff60 msgr2=0x7f10c0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:29.620 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:29.620+0000 7f10d759e700 1 --2- 172.21.15.194:0/2496481921 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f10c002ff60 0x7f10c0032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f10c80096a0 tx=0x7f10c8006b40).stop 2024-01-18T06:50:29.620 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:29.620+0000 7f10d759e700 1 -- 172.21.15.194:0/2496481921 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f10d80f62a0 msgr2=0x7f10d81064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:29.620 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:29.620+0000 7f10d759e700 1 --2- 172.21.15.194:0/2496481921 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f10d80f62a0 0x7f10d81064b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f10cc013510 tx=0x7f10cc00b040).stop 2024-01-18T06:50:29.620 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:29.620+0000 7f10d759e700 1 -- 172.21.15.194:0/2496481921 shutdown_connections 2024-01-18T06:50:29.621 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:29.620+0000 7f10d759e700 1 --2- 172.21.15.194:0/2496481921 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f10c002ff60 0x7f10c0032420 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:29.621 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:29.620+0000 7f10d759e700 1 --2- 172.21.15.194:0/2496481921 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f10d80f62a0 0x7f10d81064b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:29.621 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:29.620+0000 7f10d759e700 1 -- 172.21.15.194:0/2496481921 >> 172.21.15.194:0/2496481921 conn(0x7f10d80f1aa0 msgr2=0x7f10d80f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:50:29.621 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:29.621+0000 7f10d759e700 1 -- 172.21.15.194:0/2496481921 shutdown_connections 2024-01-18T06:50:29.621 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:29.621+0000 7f10d759e700 1 -- 172.21.15.194:0/2496481921 wait complete. 2024-01-18T06:50:29.621 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:50:30.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:29 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:50:29.619892+0000 mon.smithi047 (mon.0) 150 : audit [DBG] from='client.? 172.21.15.194:0/2496481921' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:50:31.012 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:50:31.012 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:50:31.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:30 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:50:29.992273+0000 mgr.smithi047.fuhzzr (mgr.14162) 61 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:50:31.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:30 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:50:30.112376+0000 mon.smithi047 (mon.0) 151 : audit [INF] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.fuhzzr/mirror_snapshot_schedule"}]: dispatch 2024-01-18T06:50:31.214 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:30 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:50:30.114606+0000 mon.smithi047 (mon.0) 152 : audit [INF] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.fuhzzr/trash_purge_schedule"}]: dispatch 2024-01-18T06:50:32.231 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:32.228+0000 7f0460517700 1 -- 172.21.15.194:0/3939684094 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04580f6510 msgr2=0x7f04580f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:32.232 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:32.228+0000 7f0460517700 1 --2- 172.21.15.194:0/3939684094 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04580f6510 0x7f04580f6930 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f0448009fb0 tx=0x7f0448009b30).stop 2024-01-18T06:50:32.232 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:32.228+0000 7f0460517700 1 -- 172.21.15.194:0/3939684094 shutdown_connections 2024-01-18T06:50:32.232 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:32.228+0000 7f0460517700 1 --2- 172.21.15.194:0/3939684094 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04580f6510 0x7f04580f6930 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:32.232 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:32.228+0000 7f0460517700 1 -- 172.21.15.194:0/3939684094 >> 172.21.15.194:0/3939684094 conn(0x7f04580f1c70 msgr2=0x7f04580f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:50:32.232 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:32.228+0000 7f0460517700 1 -- 172.21.15.194:0/3939684094 shutdown_connections 2024-01-18T06:50:32.232 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:32.228+0000 7f0460517700 1 -- 172.21.15.194:0/3939684094 wait complete. 2024-01-18T06:50:32.232 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:32.229+0000 7f0460517700 1 Processor -- start 2024-01-18T06:50:32.232 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:32.229+0000 7f0460517700 1 -- start start 2024-01-18T06:50:32.232 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:32.229+0000 7f0460517700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04580f6510 0x7f0458106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:50:32.232 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:32.229+0000 7f0460517700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0458106c20 con 0x7f04580f6510 2024-01-18T06:50:32.233 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:32.229+0000 7f045e2b3700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04580f6510 0x7f0458106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:50:32.233 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:32.229+0000 7f045e2b3700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04580f6510 0x7f0458106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:36896/0 (socket says 172.21.15.194:36896) 2024-01-18T06:50:32.233 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:32.229+0000 7f045e2b3700 1 -- 172.21.15.194:0/4041261489 learned_addr learned my addr 172.21.15.194:0/4041261489 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:50:32.233 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:32.230+0000 7f045e2b3700 1 -- 172.21.15.194:0/4041261489 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0448005040 con 0x7f04580f6510 2024-01-18T06:50:32.233 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:32.230+0000 7f045e2b3700 1 --2- 172.21.15.194:0/4041261489 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04580f6510 0x7f0458106730 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f0448014000 tx=0x7f044800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:50:32.233 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:32.230+0000 7f044effd700 1 -- 172.21.15.194:0/4041261489 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f04480095f0 con 0x7f04580f6510 2024-01-18T06:50:32.233 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:32.230+0000 7f044effd700 1 -- 172.21.15.194:0/4041261489 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0448007e70 con 0x7f04580f6510 2024-01-18T06:50:32.233 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:32.230+0000 7f0460517700 1 -- 172.21.15.194:0/4041261489 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0458106e20 con 0x7f04580f6510 2024-01-18T06:50:32.233 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:32.230+0000 7f044effd700 1 -- 172.21.15.194:0/4041261489 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0448014ea0 con 0x7f04580f6510 2024-01-18T06:50:32.233 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:32.230+0000 7f0460517700 1 -- 172.21.15.194:0/4041261489 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04581072b0 con 0x7f04580f6510 2024-01-18T06:50:32.233 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:32.231+0000 7f044effd700 1 -- 172.21.15.194:0/4041261489 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0448025070 con 0x7f04580f6510 2024-01-18T06:50:32.233 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:32.231+0000 7f0460517700 1 -- 172.21.15.194:0/4041261489 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0458045ad0 con 0x7f04580f6510 2024-01-18T06:50:32.233 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:32.231+0000 7f044effd700 1 --2- 172.21.15.194:0/4041261489 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f044402ffb0 0x7f0444032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:50:32.233 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:32.231+0000 7f044effd700 1 -- 172.21.15.194:0/4041261489 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0448040920 con 0x7f04580f6510 2024-01-18T06:50:32.233 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:32.231+0000 7f045dab2700 1 --2- 172.21.15.194:0/4041261489 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f044402ffb0 0x7f0444032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:50:32.234 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:32.232+0000 7f045dab2700 1 --2- 172.21.15.194:0/4041261489 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f044402ffb0 0x7f0444032470 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f04540096a0 tx=0x7f0454006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:50:32.235 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:32.235+0000 7f044effd700 1 -- 172.21.15.194:0/4041261489 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f04480079c0 con 0x7f04580f6510 2024-01-18T06:50:32.438 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:32.439+0000 7f0460517700 1 -- 172.21.15.194:0/4041261489 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f04580fbf60 con 0x7f04580f6510 2024-01-18T06:50:32.439 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:32.440+0000 7f044effd700 1 -- 172.21.15.194:0/4041261489 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f044800f300 con 0x7f04580f6510 2024-01-18T06:50:32.439 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:50:32.439 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:50:32.441 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:32.441+0000 7f0460517700 1 -- 172.21.15.194:0/4041261489 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f044402ffb0 msgr2=0x7f0444032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:32.441 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:32.441+0000 7f0460517700 1 --2- 172.21.15.194:0/4041261489 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f044402ffb0 0x7f0444032470 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f04540096a0 tx=0x7f0454006b40).stop 2024-01-18T06:50:32.441 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:32.441+0000 7f0460517700 1 -- 172.21.15.194:0/4041261489 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04580f6510 msgr2=0x7f0458106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:32.441 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:32.441+0000 7f0460517700 1 --2- 172.21.15.194:0/4041261489 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04580f6510 0x7f0458106730 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f0448014000 tx=0x7f044800b040).stop 2024-01-18T06:50:32.441 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:32.441+0000 7f0460517700 1 -- 172.21.15.194:0/4041261489 shutdown_connections 2024-01-18T06:50:32.441 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:32.441+0000 7f0460517700 1 --2- 172.21.15.194:0/4041261489 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f044402ffb0 0x7f0444032470 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:32.441 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:32.441+0000 7f0460517700 1 --2- 172.21.15.194:0/4041261489 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04580f6510 0x7f0458106730 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:32.441 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:32.441+0000 7f0460517700 1 -- 172.21.15.194:0/4041261489 >> 172.21.15.194:0/4041261489 conn(0x7f04580f1c70 msgr2=0x7f04580f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:50:32.442 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:32.441+0000 7f0460517700 1 -- 172.21.15.194:0/4041261489 shutdown_connections 2024-01-18T06:50:32.442 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:32.441+0000 7f0460517700 1 -- 172.21.15.194:0/4041261489 wait complete. 2024-01-18T06:50:32.442 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:50:33.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:33 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:50:31.992543+0000 mgr.smithi047.fuhzzr (mgr.14162) 62 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:50:33.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:33 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:50:32.440565+0000 mon.smithi047 (mon.0) 153 : audit [DBG] from='client.? 172.21.15.194:0/4041261489' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:50:33.842 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:50:33.842 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:50:35.023 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:35.019+0000 7f517f54d700 1 -- 172.21.15.194:0/1019869816 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f51780f5410 msgr2=0x7f51780f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:35.023 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:35.019+0000 7f517f54d700 1 --2- 172.21.15.194:0/1019869816 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f51780f5410 0x7f51780f5830 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f5174009fb0 tx=0x7f5174009b30).stop 2024-01-18T06:50:35.023 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:35.019+0000 7f517f54d700 1 -- 172.21.15.194:0/1019869816 shutdown_connections 2024-01-18T06:50:35.024 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:35.019+0000 7f517f54d700 1 --2- 172.21.15.194:0/1019869816 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f51780f5410 0x7f51780f5830 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:35.024 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:35.019+0000 7f517f54d700 1 -- 172.21.15.194:0/1019869816 >> 172.21.15.194:0/1019869816 conn(0x7f51780f0b70 msgr2=0x7f51780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:50:35.024 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:35.019+0000 7f517f54d700 1 -- 172.21.15.194:0/1019869816 shutdown_connections 2024-01-18T06:50:35.024 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:35.019+0000 7f517f54d700 1 -- 172.21.15.194:0/1019869816 wait complete. 2024-01-18T06:50:35.024 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:35.020+0000 7f517f54d700 1 Processor -- start 2024-01-18T06:50:35.024 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:35.020+0000 7f517f54d700 1 -- start start 2024-01-18T06:50:35.025 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:35.020+0000 7f517f54d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f51780f5410 0x7f5178109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:50:35.025 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:35.020+0000 7f517f54d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5178060240 con 0x7f51780f5410 2024-01-18T06:50:35.025 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:35.020+0000 7f517d2e9700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f51780f5410 0x7f5178109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:50:35.025 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:35.020+0000 7f517d2e9700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f51780f5410 0x7f5178109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:36904/0 (socket says 172.21.15.194:36904) 2024-01-18T06:50:35.025 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:35.020+0000 7f517d2e9700 1 -- 172.21.15.194:0/1248953771 learned_addr learned my addr 172.21.15.194:0/1248953771 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:50:35.025 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:35.021+0000 7f517d2e9700 1 -- 172.21.15.194:0/1248953771 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5174005040 con 0x7f51780f5410 2024-01-18T06:50:35.026 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:35.021+0000 7f517d2e9700 1 --2- 172.21.15.194:0/1248953771 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f51780f5410 0x7f5178109430 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f5174013510 tx=0x7f517400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:50:35.026 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:35.021+0000 7f516dffb700 1 -- 172.21.15.194:0/1248953771 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5174009320 con 0x7f51780f5410 2024-01-18T06:50:35.026 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:35.021+0000 7f516dffb700 1 -- 172.21.15.194:0/1248953771 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5174013f00 con 0x7f51780f5410 2024-01-18T06:50:35.026 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:35.021+0000 7f517f54d700 1 -- 172.21.15.194:0/1248953771 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5178060440 con 0x7f51780f5410 2024-01-18T06:50:35.026 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:35.021+0000 7f516dffb700 1 -- 172.21.15.194:0/1248953771 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5174014d90 con 0x7f51780f5410 2024-01-18T06:50:35.026 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:35.021+0000 7f517f54d700 1 -- 172.21.15.194:0/1248953771 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5178109ae0 con 0x7f51780f5410 2024-01-18T06:50:35.026 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:35.022+0000 7f517f54d700 1 -- 172.21.15.194:0/1248953771 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5178045ad0 con 0x7f51780f5410 2024-01-18T06:50:35.026 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:35.022+0000 7f516dffb700 1 -- 172.21.15.194:0/1248953771 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f51740095f0 con 0x7f51780f5410 2024-01-18T06:50:35.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:35.022+0000 7f516dffb700 1 --2- 172.21.15.194:0/1248953771 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f516402ff60 0x7f5164032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:50:35.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:35.022+0000 7f516dffb700 1 -- 172.21.15.194:0/1248953771 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f51740410c0 con 0x7f51780f5410 2024-01-18T06:50:35.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:35.023+0000 7f517cae8700 1 --2- 172.21.15.194:0/1248953771 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f516402ff60 0x7f5164032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:50:35.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:35.023+0000 7f517cae8700 1 --2- 172.21.15.194:0/1248953771 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f516402ff60 0x7f5164032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f51680096a0 tx=0x7f5168006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:50:35.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:35.027+0000 7f516dffb700 1 -- 172.21.15.194:0/1248953771 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f517401d7f0 con 0x7f51780f5410 2024-01-18T06:50:35.231 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:35.231+0000 7f517f54d700 1 -- 172.21.15.194:0/1248953771 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f51780fae60 con 0x7f51780f5410 2024-01-18T06:50:35.231 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:35.232+0000 7f516dffb700 1 -- 172.21.15.194:0/1248953771 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5174010230 con 0x7f51780f5410 2024-01-18T06:50:35.231 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:50:35.231 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:50:35.233 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:35.233+0000 7f517f54d700 1 -- 172.21.15.194:0/1248953771 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f516402ff60 msgr2=0x7f5164032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:35.233 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:35.233+0000 7f517f54d700 1 --2- 172.21.15.194:0/1248953771 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f516402ff60 0x7f5164032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f51680096a0 tx=0x7f5168006b40).stop 2024-01-18T06:50:35.233 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:35.233+0000 7f517f54d700 1 -- 172.21.15.194:0/1248953771 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f51780f5410 msgr2=0x7f5178109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:35.233 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:35.233+0000 7f517f54d700 1 --2- 172.21.15.194:0/1248953771 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f51780f5410 0x7f5178109430 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f5174013510 tx=0x7f517400b040).stop 2024-01-18T06:50:35.233 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:35.233+0000 7f517f54d700 1 -- 172.21.15.194:0/1248953771 shutdown_connections 2024-01-18T06:50:35.233 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:35.233+0000 7f517f54d700 1 --2- 172.21.15.194:0/1248953771 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f516402ff60 0x7f5164032420 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:35.233 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:35.233+0000 7f517f54d700 1 --2- 172.21.15.194:0/1248953771 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f51780f5410 0x7f5178109430 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:35.233 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:35.233+0000 7f517f54d700 1 -- 172.21.15.194:0/1248953771 >> 172.21.15.194:0/1248953771 conn(0x7f51780f0b70 msgr2=0x7f51780f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:50:35.233 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:35.233+0000 7f517f54d700 1 -- 172.21.15.194:0/1248953771 shutdown_connections 2024-01-18T06:50:35.233 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:35.233+0000 7f517f54d700 1 -- 172.21.15.194:0/1248953771 wait complete. 2024-01-18T06:50:35.234 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:50:35.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:35 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:50:33.992823+0000 mgr.smithi047.fuhzzr (mgr.14162) 63 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:50:36.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:36 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:50:35.232688+0000 mon.smithi047 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.194:0/1248953771' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:50:36.619 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:50:36.619 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:50:37.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:50:35.993076+0000 mgr.smithi047.fuhzzr (mgr.14162) 64 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:50:37.825 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:37.824+0000 7f4445597700 1 -- 172.21.15.194:0/1712973174 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44400f62a0 msgr2=0x7f44400f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:37.826 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:37.824+0000 7f4445597700 1 --2- 172.21.15.194:0/1712973174 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44400f62a0 0x7f44400f66c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f4430009fb0 tx=0x7f4430009b30).stop 2024-01-18T06:50:37.826 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:37.824+0000 7f4445597700 1 -- 172.21.15.194:0/1712973174 shutdown_connections 2024-01-18T06:50:37.826 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:37.824+0000 7f4445597700 1 --2- 172.21.15.194:0/1712973174 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44400f62a0 0x7f44400f66c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:37.826 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:37.824+0000 7f4445597700 1 -- 172.21.15.194:0/1712973174 >> 172.21.15.194:0/1712973174 conn(0x7f44400f1aa0 msgr2=0x7f44400f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:50:37.826 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:37.824+0000 7f4445597700 1 -- 172.21.15.194:0/1712973174 shutdown_connections 2024-01-18T06:50:37.826 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:37.824+0000 7f4445597700 1 -- 172.21.15.194:0/1712973174 wait complete. 2024-01-18T06:50:37.826 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:37.824+0000 7f4445597700 1 Processor -- start 2024-01-18T06:50:37.826 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:37.825+0000 7f4445597700 1 -- start start 2024-01-18T06:50:37.826 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:37.825+0000 7f4445597700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44400f62a0 0x7f444006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:50:37.826 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:37.825+0000 7f4445597700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4440069e20 con 0x7f44400f62a0 2024-01-18T06:50:37.826 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:37.825+0000 7f443ffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44400f62a0 0x7f444006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:50:37.826 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:37.825+0000 7f443ffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44400f62a0 0x7f444006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:47742/0 (socket says 172.21.15.194:47742) 2024-01-18T06:50:37.826 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:37.825+0000 7f443ffff700 1 -- 172.21.15.194:0/3308215903 learned_addr learned my addr 172.21.15.194:0/3308215903 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:50:37.828 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:37.825+0000 7f443ffff700 1 -- 172.21.15.194:0/3308215903 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4430005040 con 0x7f44400f62a0 2024-01-18T06:50:37.828 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:37.825+0000 7f443ffff700 1 --2- 172.21.15.194:0/3308215903 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44400f62a0 0x7f444006b9a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f4430013510 tx=0x7f443000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:50:37.828 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:37.826+0000 7f443cff9700 1 -- 172.21.15.194:0/3308215903 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4430009320 con 0x7f44400f62a0 2024-01-18T06:50:37.828 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:37.826+0000 7f4445597700 1 -- 172.21.15.194:0/3308215903 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4440069fc0 con 0x7f44400f62a0 2024-01-18T06:50:37.828 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:37.826+0000 7f443cff9700 1 -- 172.21.15.194:0/3308215903 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4430013f00 con 0x7f44400f62a0 2024-01-18T06:50:37.828 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:37.826+0000 7f443cff9700 1 -- 172.21.15.194:0/3308215903 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4430014dc0 con 0x7f44400f62a0 2024-01-18T06:50:37.828 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:37.826+0000 7f4445597700 1 -- 172.21.15.194:0/3308215903 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f444006a450 con 0x7f44400f62a0 2024-01-18T06:50:37.829 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:37.827+0000 7f443cff9700 1 -- 172.21.15.194:0/3308215903 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4430014420 con 0x7f44400f62a0 2024-01-18T06:50:37.829 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:37.827+0000 7f4445597700 1 -- 172.21.15.194:0/3308215903 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4440059100 con 0x7f44400f62a0 2024-01-18T06:50:37.829 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:37.827+0000 7f443cff9700 1 --2- 172.21.15.194:0/3308215903 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f442802ff60 0x7f4428032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:50:37.829 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:37.827+0000 7f443cff9700 1 -- 172.21.15.194:0/3308215903 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4430041050 con 0x7f44400f62a0 2024-01-18T06:50:37.829 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:37.827+0000 7f443f7fe700 1 --2- 172.21.15.194:0/3308215903 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f442802ff60 0x7f4428032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:50:37.829 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:37.828+0000 7f443f7fe700 1 --2- 172.21.15.194:0/3308215903 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f442802ff60 0x7f4428032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f44340096a0 tx=0x7f4434006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:50:37.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:37.832+0000 7f443cff9700 1 -- 172.21.15.194:0/3308215903 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f44300095f0 con 0x7f44400f62a0 2024-01-18T06:50:38.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:38.039+0000 7f4445597700 1 -- 172.21.15.194:0/3308215903 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f44400fbce0 con 0x7f44400f62a0 2024-01-18T06:50:38.040 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:38.039+0000 7f443cff9700 1 -- 172.21.15.194:0/3308215903 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f443001cb10 con 0x7f44400f62a0 2024-01-18T06:50:38.040 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:50:38.040 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:50:38.041 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:38.041+0000 7f4445597700 1 -- 172.21.15.194:0/3308215903 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f442802ff60 msgr2=0x7f4428032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:38.042 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:38.041+0000 7f4445597700 1 --2- 172.21.15.194:0/3308215903 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f442802ff60 0x7f4428032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f44340096a0 tx=0x7f4434006b40).stop 2024-01-18T06:50:38.042 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:38.041+0000 7f4445597700 1 -- 172.21.15.194:0/3308215903 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44400f62a0 msgr2=0x7f444006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:38.042 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:38.041+0000 7f4445597700 1 --2- 172.21.15.194:0/3308215903 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44400f62a0 0x7f444006b9a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f4430013510 tx=0x7f443000b040).stop 2024-01-18T06:50:38.042 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:38.041+0000 7f4445597700 1 -- 172.21.15.194:0/3308215903 shutdown_connections 2024-01-18T06:50:38.042 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:38.041+0000 7f4445597700 1 --2- 172.21.15.194:0/3308215903 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f442802ff60 0x7f4428032420 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:38.042 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:38.041+0000 7f4445597700 1 --2- 172.21.15.194:0/3308215903 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44400f62a0 0x7f444006b9a0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:38.042 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:38.041+0000 7f4445597700 1 -- 172.21.15.194:0/3308215903 >> 172.21.15.194:0/3308215903 conn(0x7f44400f1aa0 msgr2=0x7f44400f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:50:38.042 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:38.041+0000 7f4445597700 1 -- 172.21.15.194:0/3308215903 shutdown_connections 2024-01-18T06:50:38.042 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:38.041+0000 7f4445597700 1 -- 172.21.15.194:0/3308215903 wait complete. 2024-01-18T06:50:38.042 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:50:39.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:38 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:50:37.993358+0000 mgr.smithi047.fuhzzr (mgr.14162) 65 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:50:39.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:38 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:50:38.041324+0000 mon.smithi047 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.194:0/3308215903' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:50:39.357 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:50:39.357 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:50:40.563 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:40.561+0000 7f6b51a34700 1 -- 172.21.15.194:0/1896348143 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6b4c0f6960 msgr2=0x7f6b4c0f6d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:40.563 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:40.561+0000 7f6b51a34700 1 --2- 172.21.15.194:0/1896348143 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6b4c0f6960 0x7f6b4c0f6d80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f6b3c009fb0 tx=0x7f6b3c009b30).stop 2024-01-18T06:50:40.564 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:40.562+0000 7f6b51a34700 1 -- 172.21.15.194:0/1896348143 shutdown_connections 2024-01-18T06:50:40.564 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:40.562+0000 7f6b51a34700 1 --2- 172.21.15.194:0/1896348143 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6b4c0f6960 0x7f6b4c0f6d80 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:40.564 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:40.562+0000 7f6b51a34700 1 -- 172.21.15.194:0/1896348143 >> 172.21.15.194:0/1896348143 conn(0x7f6b4c0f0260 msgr2=0x7f6b4c0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:50:40.564 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:40.562+0000 7f6b51a34700 1 -- 172.21.15.194:0/1896348143 shutdown_connections 2024-01-18T06:50:40.564 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:40.562+0000 7f6b51a34700 1 -- 172.21.15.194:0/1896348143 wait complete. 2024-01-18T06:50:40.564 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:40.562+0000 7f6b51a34700 1 Processor -- start 2024-01-18T06:50:40.564 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:40.562+0000 7f6b51a34700 1 -- start start 2024-01-18T06:50:40.564 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:40.563+0000 7f6b51a34700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6b4c0f6960 0x7f6b4c109390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:50:40.564 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:40.563+0000 7f6b51a34700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4c0f52f0 con 0x7f6b4c0f6960 2024-01-18T06:50:40.566 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:40.563+0000 7f6b4affd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6b4c0f6960 0x7f6b4c109390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:50:40.566 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:40.563+0000 7f6b4affd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6b4c0f6960 0x7f6b4c109390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:47762/0 (socket says 172.21.15.194:47762) 2024-01-18T06:50:40.566 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:40.563+0000 7f6b4affd700 1 -- 172.21.15.194:0/2306949994 learned_addr learned my addr 172.21.15.194:0/2306949994 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:50:40.566 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:40.563+0000 7f6b4affd700 1 -- 172.21.15.194:0/2306949994 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b3c005040 con 0x7f6b4c0f6960 2024-01-18T06:50:40.566 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:40.564+0000 7f6b4affd700 1 --2- 172.21.15.194:0/2306949994 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6b4c0f6960 0x7f6b4c109390 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f6b3c013510 tx=0x7f6b3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:50:40.566 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:40.564+0000 7f6b337fe700 1 -- 172.21.15.194:0/2306949994 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b3c009320 con 0x7f6b4c0f6960 2024-01-18T06:50:40.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:40.564+0000 7f6b51a34700 1 -- 172.21.15.194:0/2306949994 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b4c0f54f0 con 0x7f6b4c0f6960 2024-01-18T06:50:40.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:40.564+0000 7f6b337fe700 1 -- 172.21.15.194:0/2306949994 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6b3c013f00 con 0x7f6b4c0f6960 2024-01-18T06:50:40.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:40.564+0000 7f6b337fe700 1 -- 172.21.15.194:0/2306949994 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b3c014d90 con 0x7f6b4c0f6960 2024-01-18T06:50:40.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:40.564+0000 7f6b51a34700 1 -- 172.21.15.194:0/2306949994 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b4c109a40 con 0x7f6b4c0f6960 2024-01-18T06:50:40.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:40.565+0000 7f6b337fe700 1 -- 172.21.15.194:0/2306949994 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6b3c0263a0 con 0x7f6b4c0f6960 2024-01-18T06:50:40.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:40.565+0000 7f6b51a34700 1 -- 172.21.15.194:0/2306949994 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b4c045ad0 con 0x7f6b4c0f6960 2024-01-18T06:50:40.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:40.565+0000 7f6b337fe700 1 --2- 172.21.15.194:0/2306949994 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f6b3402ffb0 0x7f6b34032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:50:40.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:40.565+0000 7f6b337fe700 1 -- 172.21.15.194:0/2306949994 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6b3c040f90 con 0x7f6b4c0f6960 2024-01-18T06:50:40.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:40.565+0000 7f6b4a7fc700 1 --2- 172.21.15.194:0/2306949994 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f6b3402ffb0 0x7f6b34032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:50:40.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:40.565+0000 7f6b4a7fc700 1 --2- 172.21.15.194:0/2306949994 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f6b3402ffb0 0x7f6b34032470 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f6b400096a0 tx=0x7f6b40006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:50:40.570 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:40.569+0000 7f6b337fe700 1 -- 172.21.15.194:0/2306949994 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6b3c0126f0 con 0x7f6b4c0f6960 2024-01-18T06:50:40.772 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:40.772+0000 7f6b51a34700 1 -- 172.21.15.194:0/2306949994 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6b4c10a250 con 0x7f6b4c0f6960 2024-01-18T06:50:40.773 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:40.773+0000 7f6b337fe700 1 -- 172.21.15.194:0/2306949994 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6b3c00f340 con 0x7f6b4c0f6960 2024-01-18T06:50:40.773 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:50:40.773 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:50:40.775 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:40.774+0000 7f6b51a34700 1 -- 172.21.15.194:0/2306949994 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f6b3402ffb0 msgr2=0x7f6b34032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:40.775 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:40.774+0000 7f6b51a34700 1 --2- 172.21.15.194:0/2306949994 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f6b3402ffb0 0x7f6b34032470 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f6b400096a0 tx=0x7f6b40006b40).stop 2024-01-18T06:50:40.775 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:40.774+0000 7f6b51a34700 1 -- 172.21.15.194:0/2306949994 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6b4c0f6960 msgr2=0x7f6b4c109390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:40.775 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:40.774+0000 7f6b51a34700 1 --2- 172.21.15.194:0/2306949994 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6b4c0f6960 0x7f6b4c109390 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f6b3c013510 tx=0x7f6b3c00b040).stop 2024-01-18T06:50:40.775 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:40.774+0000 7f6b51a34700 1 -- 172.21.15.194:0/2306949994 shutdown_connections 2024-01-18T06:50:40.775 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:40.774+0000 7f6b51a34700 1 --2- 172.21.15.194:0/2306949994 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f6b3402ffb0 0x7f6b34032470 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:40.775 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:40.774+0000 7f6b51a34700 1 --2- 172.21.15.194:0/2306949994 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6b4c0f6960 0x7f6b4c109390 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:40.775 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:40.774+0000 7f6b51a34700 1 -- 172.21.15.194:0/2306949994 >> 172.21.15.194:0/2306949994 conn(0x7f6b4c0f0260 msgr2=0x7f6b4c0f0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:50:40.775 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:40.774+0000 7f6b51a34700 1 -- 172.21.15.194:0/2306949994 shutdown_connections 2024-01-18T06:50:40.775 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:40.774+0000 7f6b51a34700 1 -- 172.21.15.194:0/2306949994 wait complete. 2024-01-18T06:50:40.776 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:50:41.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:41 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:50:39.993641+0000 mgr.smithi047.fuhzzr (mgr.14162) 66 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:50:41.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:41 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:50:40.774580+0000 mon.smithi047 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.194:0/2306949994' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:50:42.159 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:50:42.159 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:50:43.324 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:43.323+0000 7efe7515c700 1 -- 172.21.15.194:0/432574209 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe700f51a0 msgr2=0x7efe700f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:43.324 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:43.323+0000 7efe7515c700 1 --2- 172.21.15.194:0/432574209 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe700f51a0 0x7efe700f55c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7efe60009fb0 tx=0x7efe60009b30).stop 2024-01-18T06:50:43.324 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:43.323+0000 7efe7515c700 1 -- 172.21.15.194:0/432574209 shutdown_connections 2024-01-18T06:50:43.325 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:43.323+0000 7efe7515c700 1 --2- 172.21.15.194:0/432574209 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe700f51a0 0x7efe700f55c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:43.325 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:43.323+0000 7efe7515c700 1 -- 172.21.15.194:0/432574209 >> 172.21.15.194:0/432574209 conn(0x7efe700f09a0 msgr2=0x7efe700f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:50:43.325 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:43.323+0000 7efe7515c700 1 -- 172.21.15.194:0/432574209 shutdown_connections 2024-01-18T06:50:43.325 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:43.323+0000 7efe7515c700 1 -- 172.21.15.194:0/432574209 wait complete. 2024-01-18T06:50:43.325 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:43.323+0000 7efe7515c700 1 Processor -- start 2024-01-18T06:50:43.325 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:43.323+0000 7efe7515c700 1 -- start start 2024-01-18T06:50:43.325 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:43.323+0000 7efe7515c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe700f51a0 0x7efe701064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:50:43.325 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:43.323+0000 7efe7515c700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe70106990 con 0x7efe700f51a0 2024-01-18T06:50:43.325 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:43.324+0000 7efe6ffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe700f51a0 0x7efe701064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:50:43.325 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:43.324+0000 7efe6ffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe700f51a0 0x7efe701064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:47784/0 (socket says 172.21.15.194:47784) 2024-01-18T06:50:43.325 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:43.324+0000 7efe6ffff700 1 -- 172.21.15.194:0/1376210869 learned_addr learned my addr 172.21.15.194:0/1376210869 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:50:43.326 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:43.324+0000 7efe6ffff700 1 -- 172.21.15.194:0/1376210869 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe60005040 con 0x7efe700f51a0 2024-01-18T06:50:43.327 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:43.324+0000 7efe6ffff700 1 --2- 172.21.15.194:0/1376210869 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe700f51a0 0x7efe701064a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7efe60013510 tx=0x7efe6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:50:43.327 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:43.324+0000 7efe6cff9700 1 -- 172.21.15.194:0/1376210869 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe60009320 con 0x7efe700f51a0 2024-01-18T06:50:43.327 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:43.324+0000 7efe7515c700 1 -- 172.21.15.194:0/1376210869 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe70106b90 con 0x7efe700f51a0 2024-01-18T06:50:43.327 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:43.324+0000 7efe6cff9700 1 -- 172.21.15.194:0/1376210869 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efe60013f00 con 0x7efe700f51a0 2024-01-18T06:50:43.327 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:43.324+0000 7efe6cff9700 1 -- 172.21.15.194:0/1376210869 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe60014dc0 con 0x7efe700f51a0 2024-01-18T06:50:43.327 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:43.324+0000 7efe7515c700 1 -- 172.21.15.194:0/1376210869 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe70107020 con 0x7efe700f51a0 2024-01-18T06:50:43.327 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:43.325+0000 7efe6cff9700 1 -- 172.21.15.194:0/1376210869 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efe600095f0 con 0x7efe700f51a0 2024-01-18T06:50:43.327 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:43.325+0000 7efe7515c700 1 -- 172.21.15.194:0/1376210869 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe70045ad0 con 0x7efe700f51a0 2024-01-18T06:50:43.327 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:43.326+0000 7efe6cff9700 1 --2- 172.21.15.194:0/1376210869 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7efe5802ff60 0x7efe58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:50:43.327 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:43.326+0000 7efe6cff9700 1 -- 172.21.15.194:0/1376210869 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efe60041020 con 0x7efe700f51a0 2024-01-18T06:50:43.327 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:43.326+0000 7efe6f7fe700 1 --2- 172.21.15.194:0/1376210869 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7efe5802ff60 0x7efe58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:50:43.327 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:43.326+0000 7efe6f7fe700 1 --2- 172.21.15.194:0/1376210869 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7efe5802ff60 0x7efe58032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7efe640096a0 tx=0x7efe64006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:50:43.331 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:43.331+0000 7efe6cff9700 1 -- 172.21.15.194:0/1376210869 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efe60014420 con 0x7efe700f51a0 2024-01-18T06:50:43.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:43 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:50:41.993930+0000 mgr.smithi047.fuhzzr (mgr.14162) 67 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:50:43.535 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:43.535+0000 7efe7515c700 1 -- 172.21.15.194:0/1376210869 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efe70107e70 con 0x7efe700f51a0 2024-01-18T06:50:43.536 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:43.535+0000 7efe6cff9700 1 -- 172.21.15.194:0/1376210869 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efe6001c920 con 0x7efe700f51a0 2024-01-18T06:50:43.536 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:50:43.536 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:50:43.537 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:43.537+0000 7efe7515c700 1 -- 172.21.15.194:0/1376210869 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7efe5802ff60 msgr2=0x7efe58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:43.537 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:43.537+0000 7efe7515c700 1 --2- 172.21.15.194:0/1376210869 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7efe5802ff60 0x7efe58032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7efe640096a0 tx=0x7efe64006b40).stop 2024-01-18T06:50:43.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:43.537+0000 7efe7515c700 1 -- 172.21.15.194:0/1376210869 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe700f51a0 msgr2=0x7efe701064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:43.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:43.537+0000 7efe7515c700 1 --2- 172.21.15.194:0/1376210869 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe700f51a0 0x7efe701064a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7efe60013510 tx=0x7efe6000b040).stop 2024-01-18T06:50:43.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:43.537+0000 7efe7515c700 1 -- 172.21.15.194:0/1376210869 shutdown_connections 2024-01-18T06:50:43.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:43.537+0000 7efe7515c700 1 --2- 172.21.15.194:0/1376210869 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7efe5802ff60 0x7efe58032420 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:43.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:43.537+0000 7efe7515c700 1 --2- 172.21.15.194:0/1376210869 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efe700f51a0 0x7efe701064a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:43.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:43.537+0000 7efe7515c700 1 -- 172.21.15.194:0/1376210869 >> 172.21.15.194:0/1376210869 conn(0x7efe700f09a0 msgr2=0x7efe7005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:50:43.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:43.537+0000 7efe7515c700 1 -- 172.21.15.194:0/1376210869 shutdown_connections 2024-01-18T06:50:43.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:43.537+0000 7efe7515c700 1 -- 172.21.15.194:0/1376210869 wait complete. 2024-01-18T06:50:43.538 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:50:44.464 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:44 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:50:43.537197+0000 mon.smithi047 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.194:0/1376210869' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:50:44.936 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:50:44.936 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:50:45.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:45 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:50:43.994294+0000 mgr.smithi047.fuhzzr (mgr.14162) 68 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:50:46.198 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:46.195+0000 7fd621fb8700 1 -- 172.21.15.194:0/2922431148 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd61c060280 msgr2=0x7fd61c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:46.198 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:46.195+0000 7fd621fb8700 1 --2- 172.21.15.194:0/2922431148 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd61c060280 0x7fd61c0f90f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fd604009fb0 tx=0x7fd604009b30).stop 2024-01-18T06:50:46.198 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:46.195+0000 7fd621fb8700 1 -- 172.21.15.194:0/2922431148 shutdown_connections 2024-01-18T06:50:46.198 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:46.195+0000 7fd621fb8700 1 --2- 172.21.15.194:0/2922431148 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd61c060280 0x7fd61c0f90f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:46.198 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:46.195+0000 7fd621fb8700 1 -- 172.21.15.194:0/2922431148 >> 172.21.15.194:0/2922431148 conn(0x7fd61c0f0b70 msgr2=0x7fd61c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:50:46.198 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:46.195+0000 7fd621fb8700 1 -- 172.21.15.194:0/2922431148 shutdown_connections 2024-01-18T06:50:46.198 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:46.195+0000 7fd621fb8700 1 -- 172.21.15.194:0/2922431148 wait complete. 2024-01-18T06:50:46.199 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:46.197+0000 7fd621fb8700 1 Processor -- start 2024-01-18T06:50:46.199 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:46.197+0000 7fd621fb8700 1 -- start start 2024-01-18T06:50:46.199 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:46.197+0000 7fd621fb8700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd61c060280 0x7fd61c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:50:46.199 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:46.197+0000 7fd621fb8700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd61c106bd0 con 0x7fd61c060280 2024-01-18T06:50:46.199 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:46.197+0000 7fd61b7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd61c060280 0x7fd61c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:50:46.199 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:46.197+0000 7fd61b7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd61c060280 0x7fd61c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:47802/0 (socket says 172.21.15.194:47802) 2024-01-18T06:50:46.199 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:46.197+0000 7fd61b7fe700 1 -- 172.21.15.194:0/1955337265 learned_addr learned my addr 172.21.15.194:0/1955337265 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:50:46.199 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:46.198+0000 7fd61b7fe700 1 -- 172.21.15.194:0/1955337265 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd604005040 con 0x7fd61c060280 2024-01-18T06:50:46.199 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:46.198+0000 7fd61b7fe700 1 --2- 172.21.15.194:0/1955337265 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd61c060280 0x7fd61c1066e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fd604013510 tx=0x7fd60400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:50:46.199 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:46.198+0000 7fd613fff700 1 -- 172.21.15.194:0/1955337265 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd604009320 con 0x7fd61c060280 2024-01-18T06:50:46.199 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:46.198+0000 7fd621fb8700 1 -- 172.21.15.194:0/1955337265 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd61c106dd0 con 0x7fd61c060280 2024-01-18T06:50:46.199 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:46.198+0000 7fd613fff700 1 -- 172.21.15.194:0/1955337265 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd604013f00 con 0x7fd61c060280 2024-01-18T06:50:46.199 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:46.198+0000 7fd613fff700 1 -- 172.21.15.194:0/1955337265 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd604014dc0 con 0x7fd61c060280 2024-01-18T06:50:46.200 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:46.198+0000 7fd621fb8700 1 -- 172.21.15.194:0/1955337265 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd61c107260 con 0x7fd61c060280 2024-01-18T06:50:46.200 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:46.199+0000 7fd613fff700 1 -- 172.21.15.194:0/1955337265 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd6040095f0 con 0x7fd61c060280 2024-01-18T06:50:46.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:46.199+0000 7fd621fb8700 1 -- 172.21.15.194:0/1955337265 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd61c045ad0 con 0x7fd61c060280 2024-01-18T06:50:46.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:46.199+0000 7fd613fff700 1 --2- 172.21.15.194:0/1955337265 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fd60802ff60 0x7fd608032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:50:46.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:46.199+0000 7fd613fff700 1 -- 172.21.15.194:0/1955337265 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd604041000 con 0x7fd61c060280 2024-01-18T06:50:46.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:46.199+0000 7fd61affd700 1 --2- 172.21.15.194:0/1955337265 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fd60802ff60 0x7fd608032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:50:46.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:46.200+0000 7fd61affd700 1 --2- 172.21.15.194:0/1955337265 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fd60802ff60 0x7fd608032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fd60c0096a0 tx=0x7fd60c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:50:46.203 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:46.204+0000 7fd613fff700 1 -- 172.21.15.194:0/1955337265 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd60401c790 con 0x7fd61c060280 2024-01-18T06:50:46.409 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:46.409+0000 7fd621fb8700 1 -- 172.21.15.194:0/1955337265 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd61c1080b0 con 0x7fd61c060280 2024-01-18T06:50:46.409 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:46.410+0000 7fd613fff700 1 -- 172.21.15.194:0/1955337265 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd60401c790 con 0x7fd61c060280 2024-01-18T06:50:46.410 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:50:46.410 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:50:46.411 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:46.412+0000 7fd621fb8700 1 -- 172.21.15.194:0/1955337265 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fd60802ff60 msgr2=0x7fd608032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:46.411 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:46.412+0000 7fd621fb8700 1 --2- 172.21.15.194:0/1955337265 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fd60802ff60 0x7fd608032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fd60c0096a0 tx=0x7fd60c006b40).stop 2024-01-18T06:50:46.411 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:46.412+0000 7fd621fb8700 1 -- 172.21.15.194:0/1955337265 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd61c060280 msgr2=0x7fd61c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:46.411 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:46.412+0000 7fd621fb8700 1 --2- 172.21.15.194:0/1955337265 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd61c060280 0x7fd61c1066e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fd604013510 tx=0x7fd60400b040).stop 2024-01-18T06:50:46.411 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:46.412+0000 7fd621fb8700 1 -- 172.21.15.194:0/1955337265 shutdown_connections 2024-01-18T06:50:46.411 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:46.412+0000 7fd621fb8700 1 --2- 172.21.15.194:0/1955337265 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fd60802ff60 0x7fd608032420 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:46.412 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:46.412+0000 7fd621fb8700 1 --2- 172.21.15.194:0/1955337265 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd61c060280 0x7fd61c1066e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:46.412 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:46.412+0000 7fd621fb8700 1 -- 172.21.15.194:0/1955337265 >> 172.21.15.194:0/1955337265 conn(0x7fd61c0f0b70 msgr2=0x7fd61c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:50:46.412 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:46.412+0000 7fd621fb8700 1 -- 172.21.15.194:0/1955337265 shutdown_connections 2024-01-18T06:50:46.412 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:46.412+0000 7fd621fb8700 1 -- 172.21.15.194:0/1955337265 wait complete. 2024-01-18T06:50:46.412 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:50:47.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:47 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:50:45.994576+0000 mgr.smithi047.fuhzzr (mgr.14162) 69 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:50:47.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:47 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:50:46.411004+0000 mon.smithi047 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.194:0/1955337265' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:50:47.809 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:50:47.810 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:50:48.965 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:48.964+0000 7f096e37a700 1 -- 172.21.15.194:0/2834865946 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0968060280 msgr2=0x7f09680f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:48.965 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:48.964+0000 7f096e37a700 1 --2- 172.21.15.194:0/2834865946 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0968060280 0x7f09680f90f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f0950009fb0 tx=0x7f0950009b30).stop 2024-01-18T06:50:48.965 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:48.964+0000 7f096e37a700 1 -- 172.21.15.194:0/2834865946 shutdown_connections 2024-01-18T06:50:48.965 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:48.964+0000 7f096e37a700 1 --2- 172.21.15.194:0/2834865946 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0968060280 0x7f09680f90f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:48.965 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:48.964+0000 7f096e37a700 1 -- 172.21.15.194:0/2834865946 >> 172.21.15.194:0/2834865946 conn(0x7f09680f0b70 msgr2=0x7f09680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:50:48.965 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:48.964+0000 7f096e37a700 1 -- 172.21.15.194:0/2834865946 shutdown_connections 2024-01-18T06:50:48.965 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:48.964+0000 7f096e37a700 1 -- 172.21.15.194:0/2834865946 wait complete. 2024-01-18T06:50:48.965 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:48.964+0000 7f096e37a700 1 Processor -- start 2024-01-18T06:50:48.966 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:48.965+0000 7f096e37a700 1 -- start start 2024-01-18T06:50:48.966 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:48.965+0000 7f096e37a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0968060280 0x7f09680f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:50:48.966 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:48.965+0000 7f096e37a700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09680f4ff0 con 0x7f0968060280 2024-01-18T06:50:48.966 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:48.965+0000 7f0967fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0968060280 0x7f09680f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:50:48.966 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:48.965+0000 7f0967fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0968060280 0x7f09680f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:35588/0 (socket says 172.21.15.194:35588) 2024-01-18T06:50:48.966 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:48.965+0000 7f0967fff700 1 -- 172.21.15.194:0/4286048853 learned_addr learned my addr 172.21.15.194:0/4286048853 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:50:48.966 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:48.965+0000 7f0967fff700 1 -- 172.21.15.194:0/4286048853 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0950005040 con 0x7f0968060280 2024-01-18T06:50:48.966 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:48.965+0000 7f0967fff700 1 --2- 172.21.15.194:0/4286048853 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0968060280 0x7f09680f6b70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f0950013510 tx=0x7f095000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:50:48.966 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:48.966+0000 7f0964ff9700 1 -- 172.21.15.194:0/4286048853 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0950009320 con 0x7f0968060280 2024-01-18T06:50:48.966 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:48.966+0000 7f0964ff9700 1 -- 172.21.15.194:0/4286048853 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0950013f00 con 0x7f0968060280 2024-01-18T06:50:48.966 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:48.966+0000 7f0964ff9700 1 -- 172.21.15.194:0/4286048853 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0950014dc0 con 0x7f0968060280 2024-01-18T06:50:48.966 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:48.966+0000 7f096e37a700 1 -- 172.21.15.194:0/4286048853 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09680f5190 con 0x7f0968060280 2024-01-18T06:50:48.966 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:48.966+0000 7f096e37a700 1 -- 172.21.15.194:0/4286048853 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09680f5620 con 0x7f0968060280 2024-01-18T06:50:48.967 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:48.966+0000 7f0964ff9700 1 -- 172.21.15.194:0/4286048853 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f095001d3d0 con 0x7f0968060280 2024-01-18T06:50:48.967 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:48.966+0000 7f0964ff9700 1 --2- 172.21.15.194:0/4286048853 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f095402ff60 0x7f0954032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:50:48.967 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:48.967+0000 7f0964ff9700 1 -- 172.21.15.194:0/4286048853 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0950041050 con 0x7f0968060280 2024-01-18T06:50:48.967 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:48.967+0000 7f096e37a700 1 -- 172.21.15.194:0/4286048853 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0968059100 con 0x7f0968060280 2024-01-18T06:50:48.967 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:48.967+0000 7f09677fe700 1 --2- 172.21.15.194:0/4286048853 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f095402ff60 0x7f0954032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:50:48.968 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:48.967+0000 7f09677fe700 1 --2- 172.21.15.194:0/4286048853 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f095402ff60 0x7f0954032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f09580096a0 tx=0x7f0958006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:50:48.971 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:48.972+0000 7f0964ff9700 1 -- 172.21.15.194:0/4286048853 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0950012920 con 0x7f0968060280 2024-01-18T06:50:49.179 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:49.180+0000 7f096e37a700 1 -- 172.21.15.194:0/4286048853 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0968045ad0 con 0x7f0968060280 2024-01-18T06:50:49.180 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:49.180+0000 7f0964ff9700 1 -- 172.21.15.194:0/4286048853 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f095002b020 con 0x7f0968060280 2024-01-18T06:50:49.180 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:50:49.180 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:50:49.182 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:49.182+0000 7f096e37a700 1 -- 172.21.15.194:0/4286048853 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f095402ff60 msgr2=0x7f0954032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:49.182 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:49.182+0000 7f096e37a700 1 --2- 172.21.15.194:0/4286048853 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f095402ff60 0x7f0954032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f09580096a0 tx=0x7f0958006b40).stop 2024-01-18T06:50:49.182 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:49.182+0000 7f096e37a700 1 -- 172.21.15.194:0/4286048853 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0968060280 msgr2=0x7f09680f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:49.182 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:49.182+0000 7f096e37a700 1 --2- 172.21.15.194:0/4286048853 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0968060280 0x7f09680f6b70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f0950013510 tx=0x7f095000b040).stop 2024-01-18T06:50:49.182 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:49.182+0000 7f096e37a700 1 -- 172.21.15.194:0/4286048853 shutdown_connections 2024-01-18T06:50:49.182 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:49.182+0000 7f096e37a700 1 --2- 172.21.15.194:0/4286048853 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f095402ff60 0x7f0954032420 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:49.182 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:49.182+0000 7f096e37a700 1 --2- 172.21.15.194:0/4286048853 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0968060280 0x7f09680f6b70 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:49.182 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:49.182+0000 7f096e37a700 1 -- 172.21.15.194:0/4286048853 >> 172.21.15.194:0/4286048853 conn(0x7f09680f0b70 msgr2=0x7f09680f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:50:49.182 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:49.182+0000 7f096e37a700 1 -- 172.21.15.194:0/4286048853 shutdown_connections 2024-01-18T06:50:49.182 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:49.182+0000 7f096e37a700 1 -- 172.21.15.194:0/4286048853 wait complete. 2024-01-18T06:50:49.183 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:50:49.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:48 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:50:47.994843+0000 mgr.smithi047.fuhzzr (mgr.14162) 70 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:50:50.214 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:49 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:50:49.181335+0000 mon.smithi047 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.194:0/4286048853' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:50:50.569 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:50:50.569 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:50:51.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:50 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:50:49.995169+0000 mgr.smithi047.fuhzzr (mgr.14162) 71 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:50:51.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:51.727+0000 7fed2c0a1700 1 -- 172.21.15.194:0/1479595856 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fed240f5410 msgr2=0x7fed240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:51.729 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:51.727+0000 7fed2c0a1700 1 --2- 172.21.15.194:0/1479595856 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fed240f5410 0x7fed240f5830 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fed18009fb0 tx=0x7fed18009b30).stop 2024-01-18T06:50:51.729 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:51.727+0000 7fed2c0a1700 1 -- 172.21.15.194:0/1479595856 shutdown_connections 2024-01-18T06:50:51.729 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:51.727+0000 7fed2c0a1700 1 --2- 172.21.15.194:0/1479595856 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fed240f5410 0x7fed240f5830 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:51.729 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:51.727+0000 7fed2c0a1700 1 -- 172.21.15.194:0/1479595856 >> 172.21.15.194:0/1479595856 conn(0x7fed240f0b70 msgr2=0x7fed240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:50:51.729 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:51.727+0000 7fed2c0a1700 1 -- 172.21.15.194:0/1479595856 shutdown_connections 2024-01-18T06:50:51.729 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:51.727+0000 7fed2c0a1700 1 -- 172.21.15.194:0/1479595856 wait complete. 2024-01-18T06:50:51.729 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:51.728+0000 7fed2c0a1700 1 Processor -- start 2024-01-18T06:50:51.729 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:51.728+0000 7fed2c0a1700 1 -- start start 2024-01-18T06:50:51.729 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:51.728+0000 7fed2c0a1700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fed240f5410 0x7fed24106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:50:51.729 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:51.728+0000 7fed2c0a1700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed24106c00 con 0x7fed240f5410 2024-01-18T06:50:51.729 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:51.728+0000 7fed29e3d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fed240f5410 0x7fed24106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:50:51.730 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:51.728+0000 7fed29e3d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fed240f5410 0x7fed24106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:35602/0 (socket says 172.21.15.194:35602) 2024-01-18T06:50:51.730 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:51.728+0000 7fed29e3d700 1 -- 172.21.15.194:0/2533295079 learned_addr learned my addr 172.21.15.194:0/2533295079 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:50:51.730 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:51.729+0000 7fed29e3d700 1 -- 172.21.15.194:0/2533295079 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed18005040 con 0x7fed240f5410 2024-01-18T06:50:51.730 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:51.729+0000 7fed29e3d700 1 --2- 172.21.15.194:0/2533295079 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fed240f5410 0x7fed24106710 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fed18013510 tx=0x7fed1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:50:51.730 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:51.729+0000 7fed167fc700 1 -- 172.21.15.194:0/2533295079 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed18009320 con 0x7fed240f5410 2024-01-18T06:50:51.730 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:51.729+0000 7fed2c0a1700 1 -- 172.21.15.194:0/2533295079 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed24106e00 con 0x7fed240f5410 2024-01-18T06:50:51.730 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:51.729+0000 7fed167fc700 1 -- 172.21.15.194:0/2533295079 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fed18013f00 con 0x7fed240f5410 2024-01-18T06:50:51.730 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:51.729+0000 7fed167fc700 1 -- 172.21.15.194:0/2533295079 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed18014d90 con 0x7fed240f5410 2024-01-18T06:50:51.730 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:51.729+0000 7fed2c0a1700 1 -- 172.21.15.194:0/2533295079 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed24107290 con 0x7fed240f5410 2024-01-18T06:50:51.731 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:51.730+0000 7fed167fc700 1 -- 172.21.15.194:0/2533295079 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fed180095f0 con 0x7fed240f5410 2024-01-18T06:50:51.731 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:51.730+0000 7fed167fc700 1 --2- 172.21.15.194:0/2533295079 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fed1002ff60 0x7fed10032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:50:51.731 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:51.730+0000 7fed2c0a1700 1 -- 172.21.15.194:0/2533295079 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed24045ad0 con 0x7fed240f5410 2024-01-18T06:50:51.731 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:51.730+0000 7fed167fc700 1 -- 172.21.15.194:0/2533295079 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fed18040fe0 con 0x7fed240f5410 2024-01-18T06:50:51.731 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:51.730+0000 7fed2963c700 1 --2- 172.21.15.194:0/2533295079 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fed1002ff60 0x7fed10032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:50:51.731 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:51.730+0000 7fed2963c700 1 --2- 172.21.15.194:0/2533295079 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fed1002ff60 0x7fed10032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fed200096a0 tx=0x7fed20006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:50:51.734 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:51.735+0000 7fed167fc700 1 -- 172.21.15.194:0/2533295079 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fed1806d2b0 con 0x7fed240f5410 2024-01-18T06:50:51.942 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:51.942+0000 7fed2c0a1700 1 -- 172.21.15.194:0/2533295079 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fed24108140 con 0x7fed240f5410 2024-01-18T06:50:51.942 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:51.943+0000 7fed167fc700 1 -- 172.21.15.194:0/2533295079 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fed1801d9d0 con 0x7fed240f5410 2024-01-18T06:50:51.943 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:50:51.943 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:50:51.944 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:51.945+0000 7fed2c0a1700 1 -- 172.21.15.194:0/2533295079 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fed1002ff60 msgr2=0x7fed10032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:51.944 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:51.945+0000 7fed2c0a1700 1 --2- 172.21.15.194:0/2533295079 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fed1002ff60 0x7fed10032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fed200096a0 tx=0x7fed20006b40).stop 2024-01-18T06:50:51.944 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:51.945+0000 7fed2c0a1700 1 -- 172.21.15.194:0/2533295079 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fed240f5410 msgr2=0x7fed24106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:51.945 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:51.945+0000 7fed2c0a1700 1 --2- 172.21.15.194:0/2533295079 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fed240f5410 0x7fed24106710 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fed18013510 tx=0x7fed1800b040).stop 2024-01-18T06:50:51.945 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:51.945+0000 7fed2c0a1700 1 -- 172.21.15.194:0/2533295079 shutdown_connections 2024-01-18T06:50:51.945 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:51.945+0000 7fed2c0a1700 1 --2- 172.21.15.194:0/2533295079 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fed1002ff60 0x7fed10032420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:51.945 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:51.945+0000 7fed2c0a1700 1 --2- 172.21.15.194:0/2533295079 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fed240f5410 0x7fed24106710 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:51.945 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:51.945+0000 7fed2c0a1700 1 -- 172.21.15.194:0/2533295079 >> 172.21.15.194:0/2533295079 conn(0x7fed240f0b70 msgr2=0x7fed240f1770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:50:51.945 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:51.945+0000 7fed2c0a1700 1 -- 172.21.15.194:0/2533295079 shutdown_connections 2024-01-18T06:50:51.945 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:51.945+0000 7fed2c0a1700 1 -- 172.21.15.194:0/2533295079 wait complete. 2024-01-18T06:50:51.945 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:50:52.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:51 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:50:51.944156+0000 mon.smithi047 (mon.0) 160 : audit [DBG] from='client.? 172.21.15.194:0/2533295079' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:50:53.356 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:50:53.357 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:50:53.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:52 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:50:51.995410+0000 mgr.smithi047.fuhzzr (mgr.14162) 72 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:50:54.517 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:54.518+0000 7ffb07c0e700 1 -- 172.21.15.194:0/1684191477 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffb000f5410 msgr2=0x7ffb000f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:54.517 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:54.518+0000 7ffb07c0e700 1 --2- 172.21.15.194:0/1684191477 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffb000f5410 0x7ffb000f5830 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7ffaf0009fb0 tx=0x7ffaf0009b30).stop 2024-01-18T06:50:54.518 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:54.518+0000 7ffb07c0e700 1 -- 172.21.15.194:0/1684191477 shutdown_connections 2024-01-18T06:50:54.518 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:54.518+0000 7ffb07c0e700 1 --2- 172.21.15.194:0/1684191477 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffb000f5410 0x7ffb000f5830 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:54.518 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:54.518+0000 7ffb07c0e700 1 -- 172.21.15.194:0/1684191477 >> 172.21.15.194:0/1684191477 conn(0x7ffb000f0b70 msgr2=0x7ffb000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:50:54.518 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:54.518+0000 7ffb07c0e700 1 -- 172.21.15.194:0/1684191477 shutdown_connections 2024-01-18T06:50:54.518 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:54.518+0000 7ffb07c0e700 1 -- 172.21.15.194:0/1684191477 wait complete. 2024-01-18T06:50:54.520 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:54.518+0000 7ffb07c0e700 1 Processor -- start 2024-01-18T06:50:54.520 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:54.518+0000 7ffb07c0e700 1 -- start start 2024-01-18T06:50:54.520 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:54.518+0000 7ffb07c0e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffb000f5410 0x7ffb0010b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:50:54.520 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:54.518+0000 7ffb07c0e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb00107240 con 0x7ffb000f5410 2024-01-18T06:50:54.520 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:54.519+0000 7ffb059aa700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffb000f5410 0x7ffb0010b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:50:54.520 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:54.519+0000 7ffb059aa700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffb000f5410 0x7ffb0010b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:35618/0 (socket says 172.21.15.194:35618) 2024-01-18T06:50:54.520 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:54.519+0000 7ffb059aa700 1 -- 172.21.15.194:0/2325900800 learned_addr learned my addr 172.21.15.194:0/2325900800 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:50:54.520 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:54.519+0000 7ffb059aa700 1 -- 172.21.15.194:0/2325900800 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf0005040 con 0x7ffb000f5410 2024-01-18T06:50:54.521 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:54.519+0000 7ffb059aa700 1 --2- 172.21.15.194:0/2325900800 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffb000f5410 0x7ffb0010b850 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7ffaf0013510 tx=0x7ffaf000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:50:54.521 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:54.519+0000 7ffaf67fc700 1 -- 172.21.15.194:0/2325900800 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffaf0009320 con 0x7ffb000f5410 2024-01-18T06:50:54.521 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:54.520+0000 7ffb07c0e700 1 -- 172.21.15.194:0/2325900800 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb00060220 con 0x7ffb000f5410 2024-01-18T06:50:54.521 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:54.520+0000 7ffaf67fc700 1 -- 172.21.15.194:0/2325900800 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffaf0013f00 con 0x7ffb000f5410 2024-01-18T06:50:54.521 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:54.520+0000 7ffaf67fc700 1 -- 172.21.15.194:0/2325900800 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffaf0014d90 con 0x7ffb000f5410 2024-01-18T06:50:54.521 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:54.520+0000 7ffb07c0e700 1 -- 172.21.15.194:0/2325900800 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb0010bd40 con 0x7ffb000f5410 2024-01-18T06:50:54.521 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:54.520+0000 7ffaf67fc700 1 -- 172.21.15.194:0/2325900800 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffaf00095f0 con 0x7ffb000f5410 2024-01-18T06:50:54.521 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:54.520+0000 7ffb07c0e700 1 -- 172.21.15.194:0/2325900800 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb00045ad0 con 0x7ffb000f5410 2024-01-18T06:50:54.521 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:54.520+0000 7ffaf67fc700 1 --2- 172.21.15.194:0/2325900800 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7ffaec02ff60 0x7ffaec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:50:54.521 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:54.520+0000 7ffaf67fc700 1 -- 172.21.15.194:0/2325900800 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffaf0041000 con 0x7ffb000f5410 2024-01-18T06:50:54.521 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:54.521+0000 7ffb051a9700 1 --2- 172.21.15.194:0/2325900800 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7ffaec02ff60 0x7ffaec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:50:54.522 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:54.521+0000 7ffb051a9700 1 --2- 172.21.15.194:0/2325900800 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7ffaec02ff60 0x7ffaec032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7ffafc0096a0 tx=0x7ffafc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:50:54.524 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:54.525+0000 7ffaf67fc700 1 -- 172.21.15.194:0/2325900800 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffaf0014420 con 0x7ffb000f5410 2024-01-18T06:50:54.730 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:54.731+0000 7ffb07c0e700 1 -- 172.21.15.194:0/2325900800 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffb000fae60 con 0x7ffb000f5410 2024-01-18T06:50:54.731 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:54.731+0000 7ffaf67fc700 1 -- 172.21.15.194:0/2325900800 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ffaf0010230 con 0x7ffb000f5410 2024-01-18T06:50:54.731 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:50:54.731 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:50:54.733 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:54.733+0000 7ffb07c0e700 1 -- 172.21.15.194:0/2325900800 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7ffaec02ff60 msgr2=0x7ffaec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:54.733 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:54.733+0000 7ffb07c0e700 1 --2- 172.21.15.194:0/2325900800 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7ffaec02ff60 0x7ffaec032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7ffafc0096a0 tx=0x7ffafc006b40).stop 2024-01-18T06:50:54.733 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:54.733+0000 7ffb07c0e700 1 -- 172.21.15.194:0/2325900800 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffb000f5410 msgr2=0x7ffb0010b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:54.733 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:54.733+0000 7ffb07c0e700 1 --2- 172.21.15.194:0/2325900800 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffb000f5410 0x7ffb0010b850 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7ffaf0013510 tx=0x7ffaf000b040).stop 2024-01-18T06:50:54.733 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:54.733+0000 7ffb07c0e700 1 -- 172.21.15.194:0/2325900800 shutdown_connections 2024-01-18T06:50:54.733 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:54.733+0000 7ffb07c0e700 1 --2- 172.21.15.194:0/2325900800 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7ffaec02ff60 0x7ffaec032420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:54.733 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:54.733+0000 7ffb07c0e700 1 --2- 172.21.15.194:0/2325900800 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffb000f5410 0x7ffb0010b850 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:54.733 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:54.733+0000 7ffb07c0e700 1 -- 172.21.15.194:0/2325900800 >> 172.21.15.194:0/2325900800 conn(0x7ffb000f0b70 msgr2=0x7ffb000f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:50:54.733 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:54.733+0000 7ffb07c0e700 1 -- 172.21.15.194:0/2325900800 shutdown_connections 2024-01-18T06:50:54.734 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:54.733+0000 7ffb07c0e700 1 -- 172.21.15.194:0/2325900800 wait complete. 2024-01-18T06:50:54.734 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:50:55.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:54 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:50:53.995757+0000 mgr.smithi047.fuhzzr (mgr.14162) 73 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:50:55.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:54 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:50:54.732436+0000 mon.smithi047 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.194:0/2325900800' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:50:56.137 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:50:56.137 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:50:57.256 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:57.256+0000 7feb7b59e700 1 -- 172.21.15.194:0/3068592775 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb7c0f51a0 msgr2=0x7feb7c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:57.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:57.256+0000 7feb7b59e700 1 --2- 172.21.15.194:0/3068592775 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb7c0f51a0 0x7feb7c0f55c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7feb6c009fb0 tx=0x7feb6c009b30).stop 2024-01-18T06:50:57.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:57.256+0000 7feb7b59e700 1 -- 172.21.15.194:0/3068592775 shutdown_connections 2024-01-18T06:50:57.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:57.256+0000 7feb7b59e700 1 --2- 172.21.15.194:0/3068592775 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb7c0f51a0 0x7feb7c0f55c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:57.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:57.256+0000 7feb7b59e700 1 -- 172.21.15.194:0/3068592775 >> 172.21.15.194:0/3068592775 conn(0x7feb7c0f09a0 msgr2=0x7feb7c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:50:57.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:57.256+0000 7feb7b59e700 1 -- 172.21.15.194:0/3068592775 shutdown_connections 2024-01-18T06:50:57.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:57.256+0000 7feb7b59e700 1 -- 172.21.15.194:0/3068592775 wait complete. 2024-01-18T06:50:57.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:57.256+0000 7feb7b59e700 1 Processor -- start 2024-01-18T06:50:57.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:57.256+0000 7feb7b59e700 1 -- start start 2024-01-18T06:50:57.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:57.256+0000 7feb7b59e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb7c0f51a0 0x7feb7c109170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:50:57.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:57.256+0000 7feb7b59e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb7c060280 con 0x7feb7c0f51a0 2024-01-18T06:50:57.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:57.256+0000 7feb7a59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb7c0f51a0 0x7feb7c109170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:50:57.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:57.257+0000 7feb7a59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb7c0f51a0 0x7feb7c109170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:35628/0 (socket says 172.21.15.194:35628) 2024-01-18T06:50:57.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:57.257+0000 7feb7a59c700 1 -- 172.21.15.194:0/273949249 learned_addr learned my addr 172.21.15.194:0/273949249 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:50:57.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:57.257+0000 7feb7a59c700 1 -- 172.21.15.194:0/273949249 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb6c005040 con 0x7feb7c0f51a0 2024-01-18T06:50:57.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:57.257+0000 7feb7a59c700 1 --2- 172.21.15.194:0/273949249 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb7c0f51a0 0x7feb7c109170 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7feb6c013510 tx=0x7feb6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:50:57.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:57.257+0000 7feb6affd700 1 -- 172.21.15.194:0/273949249 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb6c009320 con 0x7feb7c0f51a0 2024-01-18T06:50:57.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:57.257+0000 7feb7b59e700 1 -- 172.21.15.194:0/273949249 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb7c060480 con 0x7feb7c0f51a0 2024-01-18T06:50:57.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:57.257+0000 7feb6affd700 1 -- 172.21.15.194:0/273949249 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feb6c013f00 con 0x7feb7c0f51a0 2024-01-18T06:50:57.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:57.257+0000 7feb6affd700 1 -- 172.21.15.194:0/273949249 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb6c014dc0 con 0x7feb7c0f51a0 2024-01-18T06:50:57.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:57.257+0000 7feb7b59e700 1 -- 172.21.15.194:0/273949249 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb7c1098a0 con 0x7feb7c0f51a0 2024-01-18T06:50:57.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:57.258+0000 7feb6affd700 1 -- 172.21.15.194:0/273949249 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feb6c0095f0 con 0x7feb7c0f51a0 2024-01-18T06:50:57.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:57.258+0000 7feb7b59e700 1 -- 172.21.15.194:0/273949249 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb7c045ad0 con 0x7feb7c0f51a0 2024-01-18T06:50:57.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:57.258+0000 7feb6affd700 1 --2- 172.21.15.194:0/273949249 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7feb6402ffb0 0x7feb64032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:50:57.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:57.258+0000 7feb6affd700 1 -- 172.21.15.194:0/273949249 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feb6c040fe0 con 0x7feb7c0f51a0 2024-01-18T06:50:57.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:57.258+0000 7feb79d9b700 1 --2- 172.21.15.194:0/273949249 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7feb6402ffb0 0x7feb64032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:50:57.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:57.259+0000 7feb79d9b700 1 --2- 172.21.15.194:0/273949249 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7feb6402ffb0 0x7feb64032470 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7feb700096a0 tx=0x7feb70006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:50:57.262 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:57.262+0000 7feb6affd700 1 -- 172.21.15.194:0/273949249 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feb6c06d2b0 con 0x7feb7c0f51a0 2024-01-18T06:50:57.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:57 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:50:55.996143+0000 mgr.smithi047.fuhzzr (mgr.14162) 74 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:50:57.470 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:57.470+0000 7feb7b59e700 1 -- 172.21.15.194:0/273949249 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feb7c0fabe0 con 0x7feb7c0f51a0 2024-01-18T06:50:57.471 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:57.471+0000 7feb6affd700 1 -- 172.21.15.194:0/273949249 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7feb6c014420 con 0x7feb7c0f51a0 2024-01-18T06:50:57.471 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:50:57.471 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:50:57.472 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:57.473+0000 7feb7b59e700 1 -- 172.21.15.194:0/273949249 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7feb6402ffb0 msgr2=0x7feb64032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:57.473 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:57.473+0000 7feb7b59e700 1 --2- 172.21.15.194:0/273949249 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7feb6402ffb0 0x7feb64032470 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7feb700096a0 tx=0x7feb70006b40).stop 2024-01-18T06:50:57.473 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:57.473+0000 7feb7b59e700 1 -- 172.21.15.194:0/273949249 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb7c0f51a0 msgr2=0x7feb7c109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:50:57.473 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:57.473+0000 7feb7b59e700 1 --2- 172.21.15.194:0/273949249 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb7c0f51a0 0x7feb7c109170 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7feb6c013510 tx=0x7feb6c00b040).stop 2024-01-18T06:50:57.473 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:57.473+0000 7feb7b59e700 1 -- 172.21.15.194:0/273949249 shutdown_connections 2024-01-18T06:50:57.473 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:57.473+0000 7feb7b59e700 1 --2- 172.21.15.194:0/273949249 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7feb6402ffb0 0x7feb64032470 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:57.473 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:57.473+0000 7feb7b59e700 1 --2- 172.21.15.194:0/273949249 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb7c0f51a0 0x7feb7c109170 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:50:57.473 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:57.473+0000 7feb7b59e700 1 -- 172.21.15.194:0/273949249 >> 172.21.15.194:0/273949249 conn(0x7feb7c0f09a0 msgr2=0x7feb7c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:50:57.473 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:57.473+0000 7feb7b59e700 1 -- 172.21.15.194:0/273949249 shutdown_connections 2024-01-18T06:50:57.473 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:50:57.473+0000 7feb7b59e700 1 -- 172.21.15.194:0/273949249 wait complete. 2024-01-18T06:50:57.473 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:50:58.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:58 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:50:57.472054+0000 mon.smithi047 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.194:0/273949249' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:50:58.888 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:50:58.889 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:50:59.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:50:59 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:50:57.996436+0000 mgr.smithi047.fuhzzr (mgr.14162) 75 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:51:00.051 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:00.049+0000 7f02e32e5700 1 -- 172.21.15.194:0/2907673807 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f02dc0f5410 msgr2=0x7f02dc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:00.051 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:00.049+0000 7f02e32e5700 1 --2- 172.21.15.194:0/2907673807 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f02dc0f5410 0x7f02dc0f5830 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f02cc009fb0 tx=0x7f02cc009b30).stop 2024-01-18T06:51:00.051 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:00.049+0000 7f02e32e5700 1 -- 172.21.15.194:0/2907673807 shutdown_connections 2024-01-18T06:51:00.051 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:00.049+0000 7f02e32e5700 1 --2- 172.21.15.194:0/2907673807 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f02dc0f5410 0x7f02dc0f5830 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:00.051 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:00.050+0000 7f02e32e5700 1 -- 172.21.15.194:0/2907673807 >> 172.21.15.194:0/2907673807 conn(0x7f02dc0f0b70 msgr2=0x7f02dc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:00.051 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:00.050+0000 7f02e32e5700 1 -- 172.21.15.194:0/2907673807 shutdown_connections 2024-01-18T06:51:00.051 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:00.050+0000 7f02e32e5700 1 -- 172.21.15.194:0/2907673807 wait complete. 2024-01-18T06:51:00.051 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:00.050+0000 7f02e32e5700 1 Processor -- start 2024-01-18T06:51:00.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:00.050+0000 7f02e32e5700 1 -- start start 2024-01-18T06:51:00.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:00.050+0000 7f02e32e5700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f02dc0f5410 0x7f02dc109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:00.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:00.050+0000 7f02e32e5700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02dc060240 con 0x7f02dc0f5410 2024-01-18T06:51:00.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:00.050+0000 7f02e1081700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f02dc0f5410 0x7f02dc109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:00.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:00.050+0000 7f02e1081700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f02dc0f5410 0x7f02dc109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:60220/0 (socket says 172.21.15.194:60220) 2024-01-18T06:51:00.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:00.050+0000 7f02e1081700 1 -- 172.21.15.194:0/949647406 learned_addr learned my addr 172.21.15.194:0/949647406 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:51:00.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:00.051+0000 7f02e1081700 1 -- 172.21.15.194:0/949647406 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02cc005040 con 0x7f02dc0f5410 2024-01-18T06:51:00.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:00.051+0000 7f02e1081700 1 --2- 172.21.15.194:0/949647406 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f02dc0f5410 0x7f02dc109400 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f02cc013510 tx=0x7f02cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:00.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:00.051+0000 7f02d1ffb700 1 -- 172.21.15.194:0/949647406 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02cc009320 con 0x7f02dc0f5410 2024-01-18T06:51:00.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:00.051+0000 7f02e32e5700 1 -- 172.21.15.194:0/949647406 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02dc060440 con 0x7f02dc0f5410 2024-01-18T06:51:00.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:00.051+0000 7f02d1ffb700 1 -- 172.21.15.194:0/949647406 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f02cc013f00 con 0x7f02dc0f5410 2024-01-18T06:51:00.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:00.051+0000 7f02d1ffb700 1 -- 172.21.15.194:0/949647406 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02cc014d90 con 0x7f02dc0f5410 2024-01-18T06:51:00.053 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:00.051+0000 7f02e32e5700 1 -- 172.21.15.194:0/949647406 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02dc109ab0 con 0x7f02dc0f5410 2024-01-18T06:51:00.054 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:00.052+0000 7f02d1ffb700 1 -- 172.21.15.194:0/949647406 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f02cc014420 con 0x7f02dc0f5410 2024-01-18T06:51:00.054 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:00.052+0000 7f02e32e5700 1 -- 172.21.15.194:0/949647406 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02dc045ad0 con 0x7f02dc0f5410 2024-01-18T06:51:00.054 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:00.052+0000 7f02d1ffb700 1 --2- 172.21.15.194:0/949647406 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f02c802ff60 0x7f02c8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:00.054 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:00.052+0000 7f02d1ffb700 1 -- 172.21.15.194:0/949647406 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f02cc0410c0 con 0x7f02dc0f5410 2024-01-18T06:51:00.054 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:00.052+0000 7f02e0880700 1 --2- 172.21.15.194:0/949647406 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f02c802ff60 0x7f02c8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:00.054 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:00.053+0000 7f02e0880700 1 --2- 172.21.15.194:0/949647406 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f02c802ff60 0x7f02c8032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f02d80096a0 tx=0x7f02d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:00.056 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:00.057+0000 7f02d1ffb700 1 -- 172.21.15.194:0/949647406 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f02cc0095f0 con 0x7f02dc0f5410 2024-01-18T06:51:00.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:00.263+0000 7f02e32e5700 1 -- 172.21.15.194:0/949647406 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f02dc0fae60 con 0x7f02dc0f5410 2024-01-18T06:51:00.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:00.263+0000 7f02d1ffb700 1 -- 172.21.15.194:0/949647406 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f02cc01db70 con 0x7f02dc0f5410 2024-01-18T06:51:00.264 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:51:00.264 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:51:00.264 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:00.265+0000 7f02e32e5700 1 -- 172.21.15.194:0/949647406 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f02c802ff60 msgr2=0x7f02c8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:00.265 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:00.265+0000 7f02e32e5700 1 --2- 172.21.15.194:0/949647406 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f02c802ff60 0x7f02c8032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f02d80096a0 tx=0x7f02d8006b40).stop 2024-01-18T06:51:00.265 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:00.265+0000 7f02e32e5700 1 -- 172.21.15.194:0/949647406 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f02dc0f5410 msgr2=0x7f02dc109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:00.265 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:00.265+0000 7f02e32e5700 1 --2- 172.21.15.194:0/949647406 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f02dc0f5410 0x7f02dc109400 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f02cc013510 tx=0x7f02cc00b040).stop 2024-01-18T06:51:00.265 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:00.265+0000 7f02e32e5700 1 -- 172.21.15.194:0/949647406 shutdown_connections 2024-01-18T06:51:00.265 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:00.265+0000 7f02e32e5700 1 --2- 172.21.15.194:0/949647406 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f02c802ff60 0x7f02c8032420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:00.265 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:00.265+0000 7f02e32e5700 1 --2- 172.21.15.194:0/949647406 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f02dc0f5410 0x7f02dc109400 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:00.265 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:00.265+0000 7f02e32e5700 1 -- 172.21.15.194:0/949647406 >> 172.21.15.194:0/949647406 conn(0x7f02dc0f0b70 msgr2=0x7f02dc0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:00.265 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:00.265+0000 7f02e32e5700 1 -- 172.21.15.194:0/949647406 shutdown_connections 2024-01-18T06:51:00.265 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:00.265+0000 7f02e32e5700 1 -- 172.21.15.194:0/949647406 wait complete. 2024-01-18T06:51:00.266 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:51:01.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:01 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:50:59.996713+0000 mgr.smithi047.fuhzzr (mgr.14162) 76 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:51:01.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:01 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:51:00.264334+0000 mon.smithi047 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.194:0/949647406' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:51:01.649 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:51:01.650 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:51:02.840 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:02.839+0000 7f8eaf131700 1 -- 172.21.15.194:0/2361789741 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ea80f6510 msgr2=0x7f8ea80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:02.840 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:02.839+0000 7f8eaf131700 1 --2- 172.21.15.194:0/2361789741 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ea80f6510 0x7f8ea80f6930 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f8e9c009fb0 tx=0x7f8e9c009b30).stop 2024-01-18T06:51:02.840 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:02.839+0000 7f8eaf131700 1 -- 172.21.15.194:0/2361789741 shutdown_connections 2024-01-18T06:51:02.840 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:02.839+0000 7f8eaf131700 1 --2- 172.21.15.194:0/2361789741 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ea80f6510 0x7f8ea80f6930 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:02.841 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:02.839+0000 7f8eaf131700 1 -- 172.21.15.194:0/2361789741 >> 172.21.15.194:0/2361789741 conn(0x7f8ea80f1c70 msgr2=0x7f8ea80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:02.841 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:02.840+0000 7f8eaf131700 1 -- 172.21.15.194:0/2361789741 shutdown_connections 2024-01-18T06:51:02.841 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:02.840+0000 7f8eaf131700 1 -- 172.21.15.194:0/2361789741 wait complete. 2024-01-18T06:51:02.841 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:02.840+0000 7f8eaf131700 1 Processor -- start 2024-01-18T06:51:02.841 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:02.840+0000 7f8eaf131700 1 -- start start 2024-01-18T06:51:02.841 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:02.841+0000 7f8eaf131700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ea80f6510 0x7f8ea8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:02.841 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:02.841+0000 7f8eaf131700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea8106c20 con 0x7f8ea80f6510 2024-01-18T06:51:02.842 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:02.842+0000 7f8eacecd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ea80f6510 0x7f8ea8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:02.842 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:02.843+0000 7f8eacecd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ea80f6510 0x7f8ea8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:60230/0 (socket says 172.21.15.194:60230) 2024-01-18T06:51:02.842 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:02.843+0000 7f8eacecd700 1 -- 172.21.15.194:0/3637876898 learned_addr learned my addr 172.21.15.194:0/3637876898 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:51:02.843 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:02.843+0000 7f8eacecd700 1 -- 172.21.15.194:0/3637876898 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e9c005040 con 0x7f8ea80f6510 2024-01-18T06:51:02.843 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:02.844+0000 7f8eacecd700 1 --2- 172.21.15.194:0/3637876898 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ea80f6510 0x7f8ea8106730 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f8e9c013510 tx=0x7f8e9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:02.843 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:02.844+0000 7f8ea57fa700 1 -- 172.21.15.194:0/3637876898 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e9c009320 con 0x7f8ea80f6510 2024-01-18T06:51:02.843 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:02.844+0000 7f8eaf131700 1 -- 172.21.15.194:0/3637876898 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ea8106e20 con 0x7f8ea80f6510 2024-01-18T06:51:02.843 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:02.844+0000 7f8ea57fa700 1 -- 172.21.15.194:0/3637876898 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8e9c013f00 con 0x7f8ea80f6510 2024-01-18T06:51:02.843 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:02.844+0000 7f8ea57fa700 1 -- 172.21.15.194:0/3637876898 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e9c014d90 con 0x7f8ea80f6510 2024-01-18T06:51:02.844 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:02.844+0000 7f8eaf131700 1 -- 172.21.15.194:0/3637876898 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ea81072b0 con 0x7f8ea80f6510 2024-01-18T06:51:02.844 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:02.845+0000 7f8eaf131700 1 -- 172.21.15.194:0/3637876898 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ea8045ad0 con 0x7f8ea80f6510 2024-01-18T06:51:02.845 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:02.845+0000 7f8ea57fa700 1 -- 172.21.15.194:0/3637876898 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8e9c0095f0 con 0x7f8ea80f6510 2024-01-18T06:51:02.845 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:02.845+0000 7f8ea57fa700 1 --2- 172.21.15.194:0/3637876898 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f8e9802ff60 0x7f8e98032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:02.845 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:02.845+0000 7f8ea57fa700 1 -- 172.21.15.194:0/3637876898 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8e9c0418a0 con 0x7f8ea80f6510 2024-01-18T06:51:02.845 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:02.845+0000 7f8ea7fff700 1 --2- 172.21.15.194:0/3637876898 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f8e9802ff60 0x7f8e98032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:02.845 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:02.846+0000 7f8ea7fff700 1 --2- 172.21.15.194:0/3637876898 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f8e9802ff60 0x7f8e98032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f8e940096a0 tx=0x7f8e94006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:02.850 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:02.851+0000 7f8ea57fa700 1 -- 172.21.15.194:0/3637876898 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8e9c01c970 con 0x7f8ea80f6510 2024-01-18T06:51:03.054 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:03.055+0000 7f8eaf131700 1 -- 172.21.15.194:0/3637876898 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8ea80fc050 con 0x7f8ea80f6510 2024-01-18T06:51:03.055 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:03.056+0000 7f8ea57fa700 1 -- 172.21.15.194:0/3637876898 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8e9c01c790 con 0x7f8ea80f6510 2024-01-18T06:51:03.055 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:51:03.056 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:51:03.057 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:03.057+0000 7f8eaf131700 1 -- 172.21.15.194:0/3637876898 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f8e9802ff60 msgr2=0x7f8e98032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:03.057 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:03.057+0000 7f8eaf131700 1 --2- 172.21.15.194:0/3637876898 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f8e9802ff60 0x7f8e98032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f8e940096a0 tx=0x7f8e94006b40).stop 2024-01-18T06:51:03.057 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:03.057+0000 7f8eaf131700 1 -- 172.21.15.194:0/3637876898 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ea80f6510 msgr2=0x7f8ea8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:03.057 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:03.057+0000 7f8eaf131700 1 --2- 172.21.15.194:0/3637876898 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ea80f6510 0x7f8ea8106730 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f8e9c013510 tx=0x7f8e9c00b040).stop 2024-01-18T06:51:03.057 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:03.057+0000 7f8eaf131700 1 -- 172.21.15.194:0/3637876898 shutdown_connections 2024-01-18T06:51:03.057 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:03.058+0000 7f8eaf131700 1 --2- 172.21.15.194:0/3637876898 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f8e9802ff60 0x7f8e98032420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:03.058 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:03.058+0000 7f8eaf131700 1 --2- 172.21.15.194:0/3637876898 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8ea80f6510 0x7f8ea8106730 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:03.058 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:03.058+0000 7f8eaf131700 1 -- 172.21.15.194:0/3637876898 >> 172.21.15.194:0/3637876898 conn(0x7f8ea80f1c70 msgr2=0x7f8ea80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:03.058 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:03.058+0000 7f8eaf131700 1 -- 172.21.15.194:0/3637876898 shutdown_connections 2024-01-18T06:51:03.058 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:03.058+0000 7f8eaf131700 1 -- 172.21.15.194:0/3637876898 wait complete. 2024-01-18T06:51:03.058 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:51:03.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:03 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:51:01.997032+0000 mgr.smithi047.fuhzzr (mgr.14162) 77 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:51:04.460 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:51:04.461 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:51:04.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:04 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:51:03.056564+0000 mon.smithi047 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.194:0/3637876898' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:51:05.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:05 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:51:03.997397+0000 mgr.smithi047.fuhzzr (mgr.14162) 78 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:51:05.616 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:05.615+0000 7fa3fe146700 1 -- 172.21.15.194:0/1190434587 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3f80f62a0 msgr2=0x7fa3f80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:05.616 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:05.615+0000 7fa3fe146700 1 --2- 172.21.15.194:0/1190434587 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3f80f62a0 0x7fa3f80f66c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fa3f4009fb0 tx=0x7fa3f4009b30).stop 2024-01-18T06:51:05.616 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:05.615+0000 7fa3fe146700 1 -- 172.21.15.194:0/1190434587 shutdown_connections 2024-01-18T06:51:05.616 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:05.615+0000 7fa3fe146700 1 --2- 172.21.15.194:0/1190434587 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3f80f62a0 0x7fa3f80f66c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:05.616 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:05.615+0000 7fa3fe146700 1 -- 172.21.15.194:0/1190434587 >> 172.21.15.194:0/1190434587 conn(0x7fa3f80f1aa0 msgr2=0x7fa3f80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:05.616 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:05.615+0000 7fa3fe146700 1 -- 172.21.15.194:0/1190434587 shutdown_connections 2024-01-18T06:51:05.616 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:05.615+0000 7fa3fe146700 1 -- 172.21.15.194:0/1190434587 wait complete. 2024-01-18T06:51:05.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:05.616+0000 7fa3fe146700 1 Processor -- start 2024-01-18T06:51:05.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:05.616+0000 7fa3fe146700 1 -- start start 2024-01-18T06:51:05.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:05.616+0000 7fa3fe146700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3f80f62a0 0x7fa3f81064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:05.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:05.616+0000 7fa3fe146700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f81069a0 con 0x7fa3f80f62a0 2024-01-18T06:51:05.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:05.616+0000 7fa3fd144700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3f80f62a0 0x7fa3f81064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:05.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:05.616+0000 7fa3fd144700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3f80f62a0 0x7fa3f81064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:60240/0 (socket says 172.21.15.194:60240) 2024-01-18T06:51:05.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:05.616+0000 7fa3fd144700 1 -- 172.21.15.194:0/179898586 learned_addr learned my addr 172.21.15.194:0/179898586 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:51:05.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:05.616+0000 7fa3fd144700 1 -- 172.21.15.194:0/179898586 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f4005040 con 0x7fa3f80f62a0 2024-01-18T06:51:05.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:05.617+0000 7fa3fd144700 1 --2- 172.21.15.194:0/179898586 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3f80f62a0 0x7fa3f81064b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fa3f4013510 tx=0x7fa3f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:05.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:05.617+0000 7fa3edffb700 1 -- 172.21.15.194:0/179898586 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3f4009320 con 0x7fa3f80f62a0 2024-01-18T06:51:05.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:05.617+0000 7fa3edffb700 1 -- 172.21.15.194:0/179898586 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa3f4013f00 con 0x7fa3f80f62a0 2024-01-18T06:51:05.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:05.617+0000 7fa3fe146700 1 -- 172.21.15.194:0/179898586 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3f8106ba0 con 0x7fa3f80f62a0 2024-01-18T06:51:05.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:05.617+0000 7fa3edffb700 1 -- 172.21.15.194:0/179898586 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3f4014dc0 con 0x7fa3f80f62a0 2024-01-18T06:51:05.619 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:05.617+0000 7fa3fe146700 1 -- 172.21.15.194:0/179898586 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3f8107030 con 0x7fa3f80f62a0 2024-01-18T06:51:05.619 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:05.618+0000 7fa3edffb700 1 -- 172.21.15.194:0/179898586 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa3f4014420 con 0x7fa3f80f62a0 2024-01-18T06:51:05.619 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:05.618+0000 7fa3fe146700 1 -- 172.21.15.194:0/179898586 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3f8045ad0 con 0x7fa3f80f62a0 2024-01-18T06:51:05.619 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:05.618+0000 7fa3edffb700 1 --2- 172.21.15.194:0/179898586 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa3e402ff60 0x7fa3e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:05.619 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:05.618+0000 7fa3edffb700 1 -- 172.21.15.194:0/179898586 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa3f40410f0 con 0x7fa3f80f62a0 2024-01-18T06:51:05.619 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:05.618+0000 7fa3fc943700 1 --2- 172.21.15.194:0/179898586 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa3e402ff60 0x7fa3e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:05.619 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:05.618+0000 7fa3fc943700 1 --2- 172.21.15.194:0/179898586 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa3e402ff60 0x7fa3e4032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fa3e80096a0 tx=0x7fa3e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:05.622 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:05.623+0000 7fa3edffb700 1 -- 172.21.15.194:0/179898586 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa3f40095f0 con 0x7fa3f80f62a0 2024-01-18T06:51:05.830 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:05.830+0000 7fa3fe146700 1 -- 172.21.15.194:0/179898586 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa3f8107ee0 con 0x7fa3f80f62a0 2024-01-18T06:51:05.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:05.831+0000 7fa3edffb700 1 -- 172.21.15.194:0/179898586 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa3f401cb10 con 0x7fa3f80f62a0 2024-01-18T06:51:05.831 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:51:05.831 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:51:05.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:05.833+0000 7fa3fe146700 1 -- 172.21.15.194:0/179898586 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa3e402ff60 msgr2=0x7fa3e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:05.833 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:05.833+0000 7fa3fe146700 1 --2- 172.21.15.194:0/179898586 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa3e402ff60 0x7fa3e4032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fa3e80096a0 tx=0x7fa3e8006b40).stop 2024-01-18T06:51:05.833 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:05.833+0000 7fa3fe146700 1 -- 172.21.15.194:0/179898586 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3f80f62a0 msgr2=0x7fa3f81064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:05.833 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:05.833+0000 7fa3fe146700 1 --2- 172.21.15.194:0/179898586 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3f80f62a0 0x7fa3f81064b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fa3f4013510 tx=0x7fa3f400b040).stop 2024-01-18T06:51:05.833 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:05.833+0000 7fa3fe146700 1 -- 172.21.15.194:0/179898586 shutdown_connections 2024-01-18T06:51:05.833 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:05.833+0000 7fa3fe146700 1 --2- 172.21.15.194:0/179898586 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa3e402ff60 0x7fa3e4032420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:05.833 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:05.833+0000 7fa3fe146700 1 --2- 172.21.15.194:0/179898586 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3f80f62a0 0x7fa3f81064b0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:05.833 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:05.833+0000 7fa3fe146700 1 -- 172.21.15.194:0/179898586 >> 172.21.15.194:0/179898586 conn(0x7fa3f80f1aa0 msgr2=0x7fa3f80f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:05.833 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:05.833+0000 7fa3fe146700 1 -- 172.21.15.194:0/179898586 shutdown_connections 2024-01-18T06:51:05.833 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:05.833+0000 7fa3fe146700 1 -- 172.21.15.194:0/179898586 wait complete. 2024-01-18T06:51:05.833 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:51:06.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:06 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:51:05.832042+0000 mon.smithi047 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.194:0/179898586' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:51:07.225 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:51:07.226 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:51:07.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:07 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:51:05.997772+0000 mgr.smithi047.fuhzzr (mgr.14162) 79 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:51:08.413 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:08.413+0000 7f49a4c88700 1 -- 172.21.15.194:0/3079926411 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49a00601e0 msgr2=0x7f49a00f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:08.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:08.413+0000 7f49a4c88700 1 --2- 172.21.15.194:0/3079926411 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49a00601e0 0x7f49a00f8f20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f4988009fb0 tx=0x7f4988009b30).stop 2024-01-18T06:51:08.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:08.413+0000 7f49a4c88700 1 -- 172.21.15.194:0/3079926411 shutdown_connections 2024-01-18T06:51:08.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:08.413+0000 7f49a4c88700 1 --2- 172.21.15.194:0/3079926411 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49a00601e0 0x7f49a00f8f20 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:08.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:08.413+0000 7f49a4c88700 1 -- 172.21.15.194:0/3079926411 >> 172.21.15.194:0/3079926411 conn(0x7f49a00f09a0 msgr2=0x7f49a00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:08.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:08.413+0000 7f49a4c88700 1 -- 172.21.15.194:0/3079926411 shutdown_connections 2024-01-18T06:51:08.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:08.413+0000 7f49a4c88700 1 -- 172.21.15.194:0/3079926411 wait complete. 2024-01-18T06:51:08.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:08.413+0000 7f49a4c88700 1 Processor -- start 2024-01-18T06:51:08.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:08.414+0000 7f49a4c88700 1 -- start start 2024-01-18T06:51:08.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:08.414+0000 7f49a4c88700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49a00601e0 0x7f49a00f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:08.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:08.414+0000 7f49a4c88700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a00f4e20 con 0x7f49a00601e0 2024-01-18T06:51:08.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:08.414+0000 7f499f7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49a00601e0 0x7f49a00f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:08.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:08.414+0000 7f499f7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49a00601e0 0x7f49a00f69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:45008/0 (socket says 172.21.15.194:45008) 2024-01-18T06:51:08.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:08.414+0000 7f499f7fe700 1 -- 172.21.15.194:0/2207700018 learned_addr learned my addr 172.21.15.194:0/2207700018 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:51:08.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:08.414+0000 7f499f7fe700 1 -- 172.21.15.194:0/2207700018 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4988005040 con 0x7f49a00601e0 2024-01-18T06:51:08.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:08.414+0000 7f499f7fe700 1 --2- 172.21.15.194:0/2207700018 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49a00601e0 0x7f49a00f69a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f4988013510 tx=0x7f498800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:08.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:08.415+0000 7f4997fff700 1 -- 172.21.15.194:0/2207700018 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4988009320 con 0x7f49a00601e0 2024-01-18T06:51:08.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:08.415+0000 7f49a4c88700 1 -- 172.21.15.194:0/2207700018 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49a00f4fc0 con 0x7f49a00601e0 2024-01-18T06:51:08.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:08.415+0000 7f4997fff700 1 -- 172.21.15.194:0/2207700018 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4988013f00 con 0x7f49a00601e0 2024-01-18T06:51:08.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:08.415+0000 7f4997fff700 1 -- 172.21.15.194:0/2207700018 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4988014d90 con 0x7f49a00601e0 2024-01-18T06:51:08.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:08.415+0000 7f49a4c88700 1 -- 172.21.15.194:0/2207700018 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49a00f5450 con 0x7f49a00601e0 2024-01-18T06:51:08.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:08.416+0000 7f4997fff700 1 -- 172.21.15.194:0/2207700018 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f49880095f0 con 0x7f49a00601e0 2024-01-18T06:51:08.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:08.416+0000 7f49a4c88700 1 -- 172.21.15.194:0/2207700018 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49a0059100 con 0x7f49a00601e0 2024-01-18T06:51:08.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:08.416+0000 7f4997fff700 1 --2- 172.21.15.194:0/2207700018 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f498c02ff60 0x7f498c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:08.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:08.416+0000 7f4997fff700 1 -- 172.21.15.194:0/2207700018 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4988041000 con 0x7f49a00601e0 2024-01-18T06:51:08.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:08.416+0000 7f499effd700 1 --2- 172.21.15.194:0/2207700018 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f498c02ff60 0x7f498c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:08.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:08.416+0000 7f499effd700 1 --2- 172.21.15.194:0/2207700018 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f498c02ff60 0x7f498c032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f49980096a0 tx=0x7f4998006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:08.420 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:08.420+0000 7f4997fff700 1 -- 172.21.15.194:0/2207700018 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f498801d7f0 con 0x7f49a00601e0 2024-01-18T06:51:08.625 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:08.625+0000 7f49a4c88700 1 -- 172.21.15.194:0/2207700018 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f49a00fbcd0 con 0x7f49a00601e0 2024-01-18T06:51:08.625 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:08.626+0000 7f4997fff700 1 -- 172.21.15.194:0/2207700018 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4988010230 con 0x7f49a00601e0 2024-01-18T06:51:08.625 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:51:08.626 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:51:08.627 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:08.627+0000 7f49a4c88700 1 -- 172.21.15.194:0/2207700018 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f498c02ff60 msgr2=0x7f498c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:08.627 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:08.627+0000 7f49a4c88700 1 --2- 172.21.15.194:0/2207700018 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f498c02ff60 0x7f498c032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f49980096a0 tx=0x7f4998006b40).stop 2024-01-18T06:51:08.627 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:08.627+0000 7f49a4c88700 1 -- 172.21.15.194:0/2207700018 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49a00601e0 msgr2=0x7f49a00f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:08.627 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:08.627+0000 7f49a4c88700 1 --2- 172.21.15.194:0/2207700018 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49a00601e0 0x7f49a00f69a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f4988013510 tx=0x7f498800b040).stop 2024-01-18T06:51:08.627 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:08.627+0000 7f49a4c88700 1 -- 172.21.15.194:0/2207700018 shutdown_connections 2024-01-18T06:51:08.628 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:08.628+0000 7f49a4c88700 1 --2- 172.21.15.194:0/2207700018 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f498c02ff60 0x7f498c032420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:08.628 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:08.628+0000 7f49a4c88700 1 --2- 172.21.15.194:0/2207700018 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49a00601e0 0x7f49a00f69a0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:08.628 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:08.628+0000 7f49a4c88700 1 -- 172.21.15.194:0/2207700018 >> 172.21.15.194:0/2207700018 conn(0x7f49a00f09a0 msgr2=0x7f49a00f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:08.628 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:08.628+0000 7f49a4c88700 1 -- 172.21.15.194:0/2207700018 shutdown_connections 2024-01-18T06:51:08.628 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:08.628+0000 7f49a4c88700 1 -- 172.21.15.194:0/2207700018 wait complete. 2024-01-18T06:51:08.628 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:51:09.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:08 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:51:07.998074+0000 mgr.smithi047.fuhzzr (mgr.14162) 80 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:51:09.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:08 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:51:08.627011+0000 mon.smithi047 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.194:0/2207700018' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:51:10.021 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:51:10.021 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:51:11.228 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:11.229+0000 7f963c6a9700 1 -- 172.21.15.194:0/2387566782 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96340f6510 msgr2=0x7f96340f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:11.228 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:11.229+0000 7f963c6a9700 1 --2- 172.21.15.194:0/2387566782 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96340f6510 0x7f96340f6930 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f9624009fb0 tx=0x7f9624009b30).stop 2024-01-18T06:51:11.229 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:11.229+0000 7f963c6a9700 1 -- 172.21.15.194:0/2387566782 shutdown_connections 2024-01-18T06:51:11.229 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:11.229+0000 7f963c6a9700 1 --2- 172.21.15.194:0/2387566782 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96340f6510 0x7f96340f6930 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:11.229 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:11.229+0000 7f963c6a9700 1 -- 172.21.15.194:0/2387566782 >> 172.21.15.194:0/2387566782 conn(0x7f96340f1c70 msgr2=0x7f96340f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:11.229 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:11.229+0000 7f963c6a9700 1 -- 172.21.15.194:0/2387566782 shutdown_connections 2024-01-18T06:51:11.229 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:11.229+0000 7f963c6a9700 1 -- 172.21.15.194:0/2387566782 wait complete. 2024-01-18T06:51:11.230 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:11.229+0000 7f963c6a9700 1 Processor -- start 2024-01-18T06:51:11.230 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:11.229+0000 7f963c6a9700 1 -- start start 2024-01-18T06:51:11.230 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:11.230+0000 7f963c6a9700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96340f6510 0x7f9634106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:11.230 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:11.230+0000 7f963c6a9700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9634106c20 con 0x7f96340f6510 2024-01-18T06:51:11.230 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:11.230+0000 7f963a445700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96340f6510 0x7f9634106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:11.231 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:11.230+0000 7f963a445700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96340f6510 0x7f9634106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:45018/0 (socket says 172.21.15.194:45018) 2024-01-18T06:51:11.231 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:11.230+0000 7f963a445700 1 -- 172.21.15.194:0/2285047195 learned_addr learned my addr 172.21.15.194:0/2285047195 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:51:11.231 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:11.230+0000 7f963a445700 1 -- 172.21.15.194:0/2285047195 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9624005040 con 0x7f96340f6510 2024-01-18T06:51:11.231 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:11.230+0000 7f963a445700 1 --2- 172.21.15.194:0/2285047195 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96340f6510 0x7f9634106730 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f9624013510 tx=0x7f962400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:11.231 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:11.231+0000 7f962affd700 1 -- 172.21.15.194:0/2285047195 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9624009320 con 0x7f96340f6510 2024-01-18T06:51:11.231 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:11.231+0000 7f963c6a9700 1 -- 172.21.15.194:0/2285047195 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9634106e20 con 0x7f96340f6510 2024-01-18T06:51:11.231 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:11.231+0000 7f963c6a9700 1 -- 172.21.15.194:0/2285047195 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96341072b0 con 0x7f96340f6510 2024-01-18T06:51:11.231 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:11.231+0000 7f962affd700 1 -- 172.21.15.194:0/2285047195 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9624013f00 con 0x7f96340f6510 2024-01-18T06:51:11.231 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:11.231+0000 7f962affd700 1 -- 172.21.15.194:0/2285047195 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9624014d60 con 0x7f96340f6510 2024-01-18T06:51:11.232 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:11.232+0000 7f962affd700 1 -- 172.21.15.194:0/2285047195 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f96240095f0 con 0x7f96340f6510 2024-01-18T06:51:11.232 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:11.232+0000 7f963c6a9700 1 -- 172.21.15.194:0/2285047195 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9634045ad0 con 0x7f96340f6510 2024-01-18T06:51:11.232 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:11.232+0000 7f962affd700 1 --2- 172.21.15.194:0/2285047195 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f962002ff60 0x7f9620032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:11.232 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:11.232+0000 7f962affd700 1 -- 172.21.15.194:0/2285047195 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9624041010 con 0x7f96340f6510 2024-01-18T06:51:11.232 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:11.232+0000 7f9639c44700 1 --2- 172.21.15.194:0/2285047195 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f962002ff60 0x7f9620032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:11.232 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:11.232+0000 7f9639c44700 1 --2- 172.21.15.194:0/2285047195 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f962002ff60 0x7f9620032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f96300096a0 tx=0x7f9630006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:11.236 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:11.236+0000 7f962affd700 1 -- 172.21.15.194:0/2285047195 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9624014420 con 0x7f96340f6510 2024-01-18T06:51:11.442 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:11.443+0000 7f963c6a9700 1 -- 172.21.15.194:0/2285047195 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f96340fbf60 con 0x7f96340f6510 2024-01-18T06:51:11.443 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:11.443+0000 7f962affd700 1 -- 172.21.15.194:0/2285047195 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9624012980 con 0x7f96340f6510 2024-01-18T06:51:11.443 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:51:11.443 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:51:11.445 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:11.445+0000 7f963c6a9700 1 -- 172.21.15.194:0/2285047195 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f962002ff60 msgr2=0x7f9620032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:11.445 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:11.445+0000 7f963c6a9700 1 --2- 172.21.15.194:0/2285047195 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f962002ff60 0x7f9620032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f96300096a0 tx=0x7f9630006b40).stop 2024-01-18T06:51:11.445 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:11.445+0000 7f963c6a9700 1 -- 172.21.15.194:0/2285047195 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96340f6510 msgr2=0x7f9634106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:11.445 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:11.445+0000 7f963c6a9700 1 --2- 172.21.15.194:0/2285047195 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96340f6510 0x7f9634106730 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f9624013510 tx=0x7f962400b040).stop 2024-01-18T06:51:11.445 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:11.445+0000 7f963c6a9700 1 -- 172.21.15.194:0/2285047195 shutdown_connections 2024-01-18T06:51:11.445 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:11.445+0000 7f963c6a9700 1 --2- 172.21.15.194:0/2285047195 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f962002ff60 0x7f9620032420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:11.445 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:11.445+0000 7f963c6a9700 1 --2- 172.21.15.194:0/2285047195 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96340f6510 0x7f9634106730 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:11.445 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:11.445+0000 7f963c6a9700 1 -- 172.21.15.194:0/2285047195 >> 172.21.15.194:0/2285047195 conn(0x7f96340f1c70 msgr2=0x7f96340f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:11.445 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:11.445+0000 7f963c6a9700 1 -- 172.21.15.194:0/2285047195 shutdown_connections 2024-01-18T06:51:11.445 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:11.445+0000 7f963c6a9700 1 -- 172.21.15.194:0/2285047195 wait complete. 2024-01-18T06:51:11.446 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:51:11.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:11 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:51:09.998421+0000 mgr.smithi047.fuhzzr (mgr.14162) 81 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:51:12.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:12 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:51:11.444321+0000 mon.smithi047 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.194:0/2285047195' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:51:12.840 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:51:12.841 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:51:13.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:13 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:51:11.998808+0000 mgr.smithi047.fuhzzr (mgr.14162) 82 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:51:14.098 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:14.097+0000 7fbecb7e2700 1 -- 172.21.15.194:0/3233117522 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbec40f5410 msgr2=0x7fbec40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:14.098 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:14.097+0000 7fbecb7e2700 1 --2- 172.21.15.194:0/3233117522 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbec40f5410 0x7fbec40f5830 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fbeb8009fb0 tx=0x7fbeb8009b30).stop 2024-01-18T06:51:14.098 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:14.097+0000 7fbecb7e2700 1 -- 172.21.15.194:0/3233117522 shutdown_connections 2024-01-18T06:51:14.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:14.097+0000 7fbecb7e2700 1 --2- 172.21.15.194:0/3233117522 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbec40f5410 0x7fbec40f5830 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:14.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:14.097+0000 7fbecb7e2700 1 -- 172.21.15.194:0/3233117522 >> 172.21.15.194:0/3233117522 conn(0x7fbec40f0b70 msgr2=0x7fbec40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:14.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:14.097+0000 7fbecb7e2700 1 -- 172.21.15.194:0/3233117522 shutdown_connections 2024-01-18T06:51:14.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:14.097+0000 7fbecb7e2700 1 -- 172.21.15.194:0/3233117522 wait complete. 2024-01-18T06:51:14.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:14.098+0000 7fbecb7e2700 1 Processor -- start 2024-01-18T06:51:14.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:14.098+0000 7fbecb7e2700 1 -- start start 2024-01-18T06:51:14.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:14.098+0000 7fbecb7e2700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbec40f5410 0x7fbec4109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:14.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:14.098+0000 7fbecb7e2700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec4060240 con 0x7fbec40f5410 2024-01-18T06:51:14.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:14.098+0000 7fbec957e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbec40f5410 0x7fbec4109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:14.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:14.098+0000 7fbec957e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbec40f5410 0x7fbec4109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:45044/0 (socket says 172.21.15.194:45044) 2024-01-18T06:51:14.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:14.098+0000 7fbec957e700 1 -- 172.21.15.194:0/3181131163 learned_addr learned my addr 172.21.15.194:0/3181131163 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:51:14.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:14.099+0000 7fbec957e700 1 -- 172.21.15.194:0/3181131163 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeb8005040 con 0x7fbec40f5410 2024-01-18T06:51:14.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:14.099+0000 7fbec957e700 1 --2- 172.21.15.194:0/3181131163 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbec40f5410 0x7fbec4109430 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fbeb8014000 tx=0x7fbeb800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:14.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:14.099+0000 7fbeb5ffb700 1 -- 172.21.15.194:0/3181131163 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbeb80095f0 con 0x7fbec40f5410 2024-01-18T06:51:14.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:14.099+0000 7fbecb7e2700 1 -- 172.21.15.194:0/3181131163 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbec4060440 con 0x7fbec40f5410 2024-01-18T06:51:14.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:14.099+0000 7fbeb5ffb700 1 -- 172.21.15.194:0/3181131163 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbeb8007e70 con 0x7fbec40f5410 2024-01-18T06:51:14.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:14.099+0000 7fbeb5ffb700 1 -- 172.21.15.194:0/3181131163 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbeb8014ea0 con 0x7fbec40f5410 2024-01-18T06:51:14.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:14.099+0000 7fbecb7e2700 1 -- 172.21.15.194:0/3181131163 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbec4109ae0 con 0x7fbec40f5410 2024-01-18T06:51:14.101 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:14.100+0000 7fbeb5ffb700 1 -- 172.21.15.194:0/3181131163 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbeb8025070 con 0x7fbec40f5410 2024-01-18T06:51:14.101 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:14.100+0000 7fbecb7e2700 1 -- 172.21.15.194:0/3181131163 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbec4045ad0 con 0x7fbec40f5410 2024-01-18T06:51:14.101 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:14.100+0000 7fbeb5ffb700 1 --2- 172.21.15.194:0/3181131163 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fbeb002ff60 0x7fbeb0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:14.101 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:14.100+0000 7fbeb5ffb700 1 -- 172.21.15.194:0/3181131163 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbeb8040930 con 0x7fbec40f5410 2024-01-18T06:51:14.101 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:14.100+0000 7fbec8d7d700 1 --2- 172.21.15.194:0/3181131163 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fbeb002ff60 0x7fbeb0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:14.101 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:14.101+0000 7fbec8d7d700 1 --2- 172.21.15.194:0/3181131163 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fbeb002ff60 0x7fbeb0032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fbec00096a0 tx=0x7fbec0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:14.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:14.105+0000 7fbeb5ffb700 1 -- 172.21.15.194:0/3181131163 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbeb800f4e0 con 0x7fbec40f5410 2024-01-18T06:51:14.307 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:14.307+0000 7fbecb7e2700 1 -- 172.21.15.194:0/3181131163 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbec40fae60 con 0x7fbec40f5410 2024-01-18T06:51:14.307 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:14.308+0000 7fbeb5ffb700 1 -- 172.21.15.194:0/3181131163 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbeb800f300 con 0x7fbec40f5410 2024-01-18T06:51:14.307 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:51:14.308 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:51:14.309 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:14.309+0000 7fbecb7e2700 1 -- 172.21.15.194:0/3181131163 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fbeb002ff60 msgr2=0x7fbeb0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:14.309 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:14.309+0000 7fbecb7e2700 1 --2- 172.21.15.194:0/3181131163 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fbeb002ff60 0x7fbeb0032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fbec00096a0 tx=0x7fbec0006b40).stop 2024-01-18T06:51:14.309 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:14.309+0000 7fbecb7e2700 1 -- 172.21.15.194:0/3181131163 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbec40f5410 msgr2=0x7fbec4109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:14.309 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:14.309+0000 7fbecb7e2700 1 --2- 172.21.15.194:0/3181131163 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbec40f5410 0x7fbec4109430 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fbeb8014000 tx=0x7fbeb800b040).stop 2024-01-18T06:51:14.309 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:14.309+0000 7fbecb7e2700 1 -- 172.21.15.194:0/3181131163 shutdown_connections 2024-01-18T06:51:14.309 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:14.309+0000 7fbecb7e2700 1 --2- 172.21.15.194:0/3181131163 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fbeb002ff60 0x7fbeb0032420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:14.310 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:14.309+0000 7fbecb7e2700 1 --2- 172.21.15.194:0/3181131163 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbec40f5410 0x7fbec4109430 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:14.310 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:14.309+0000 7fbecb7e2700 1 -- 172.21.15.194:0/3181131163 >> 172.21.15.194:0/3181131163 conn(0x7fbec40f0b70 msgr2=0x7fbec40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:14.310 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:14.309+0000 7fbecb7e2700 1 -- 172.21.15.194:0/3181131163 shutdown_connections 2024-01-18T06:51:14.310 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:14.309+0000 7fbecb7e2700 1 -- 172.21.15.194:0/3181131163 wait complete. 2024-01-18T06:51:14.310 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:51:15.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:15 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:51:13.999160+0000 mgr.smithi047.fuhzzr (mgr.14162) 83 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:51:15.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:15 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:51:14.308589+0000 mon.smithi047 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.194:0/3181131163' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:51:15.704 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:51:15.704 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:51:16.974 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:16.973+0000 7f0bd2631700 1 -- 172.21.15.194:0/575991082 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0bcc0f5410 msgr2=0x7f0bcc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:16.974 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:16.973+0000 7f0bd2631700 1 --2- 172.21.15.194:0/575991082 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0bcc0f5410 0x7f0bcc0f5830 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f0bbc009fb0 tx=0x7f0bbc009b30).stop 2024-01-18T06:51:16.974 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:16.973+0000 7f0bd2631700 1 -- 172.21.15.194:0/575991082 shutdown_connections 2024-01-18T06:51:16.974 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:16.973+0000 7f0bd2631700 1 --2- 172.21.15.194:0/575991082 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0bcc0f5410 0x7f0bcc0f5830 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:16.974 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:16.973+0000 7f0bd2631700 1 -- 172.21.15.194:0/575991082 >> 172.21.15.194:0/575991082 conn(0x7f0bcc0f0b70 msgr2=0x7f0bcc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:16.974 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:16.973+0000 7f0bd2631700 1 -- 172.21.15.194:0/575991082 shutdown_connections 2024-01-18T06:51:16.974 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:16.973+0000 7f0bd2631700 1 -- 172.21.15.194:0/575991082 wait complete. 2024-01-18T06:51:16.975 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:16.974+0000 7f0bd2631700 1 Processor -- start 2024-01-18T06:51:16.975 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:16.974+0000 7f0bd2631700 1 -- start start 2024-01-18T06:51:16.975 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:16.974+0000 7f0bd2631700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0bcc0f5410 0x7f0bcc1055d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:16.975 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:16.974+0000 7f0bd2631700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bcc105ac0 con 0x7f0bcc0f5410 2024-01-18T06:51:16.975 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:16.975+0000 7f0bcbfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0bcc0f5410 0x7f0bcc1055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:16.975 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:16.975+0000 7f0bcbfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0bcc0f5410 0x7f0bcc1055d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:45070/0 (socket says 172.21.15.194:45070) 2024-01-18T06:51:16.975 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:16.975+0000 7f0bcbfff700 1 -- 172.21.15.194:0/1111540322 learned_addr learned my addr 172.21.15.194:0/1111540322 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:51:16.976 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:16.975+0000 7f0bcbfff700 1 -- 172.21.15.194:0/1111540322 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bbc005040 con 0x7f0bcc0f5410 2024-01-18T06:51:16.977 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:16.975+0000 7f0bcbfff700 1 --2- 172.21.15.194:0/1111540322 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0bcc0f5410 0x7f0bcc1055d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f0bbc014000 tx=0x7f0bbc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:16.977 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:16.975+0000 7f0bc8ff9700 1 -- 172.21.15.194:0/1111540322 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bbc0095f0 con 0x7f0bcc0f5410 2024-01-18T06:51:16.977 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:16.975+0000 7f0bc8ff9700 1 -- 172.21.15.194:0/1111540322 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0bbc007e70 con 0x7f0bcc0f5410 2024-01-18T06:51:16.977 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:16.975+0000 7f0bc8ff9700 1 -- 172.21.15.194:0/1111540322 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bbc014ea0 con 0x7f0bcc0f5410 2024-01-18T06:51:16.977 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:16.975+0000 7f0bd2631700 1 -- 172.21.15.194:0/1111540322 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bcc105cc0 con 0x7f0bcc0f5410 2024-01-18T06:51:16.977 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:16.975+0000 7f0bd2631700 1 -- 172.21.15.194:0/1111540322 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bcc106150 con 0x7f0bcc0f5410 2024-01-18T06:51:16.977 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:16.976+0000 7f0bc8ff9700 1 -- 172.21.15.194:0/1111540322 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0bbc025070 con 0x7f0bcc0f5410 2024-01-18T06:51:16.977 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:16.976+0000 7f0bd2631700 1 -- 172.21.15.194:0/1111540322 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bcc045ad0 con 0x7f0bcc0f5410 2024-01-18T06:51:16.977 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:16.976+0000 7f0bc8ff9700 1 --2- 172.21.15.194:0/1111540322 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f0bb402ff60 0x7f0bb4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:16.977 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:16.976+0000 7f0bc8ff9700 1 -- 172.21.15.194:0/1111540322 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0bbc040c70 con 0x7f0bcc0f5410 2024-01-18T06:51:16.977 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:16.977+0000 7f0bcb7fe700 1 --2- 172.21.15.194:0/1111540322 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f0bb402ff60 0x7f0bb4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:16.977 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:16.977+0000 7f0bcb7fe700 1 --2- 172.21.15.194:0/1111540322 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f0bb402ff60 0x7f0bb4032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f0bc00096a0 tx=0x7f0bc0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:16.980 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:16.981+0000 7f0bc8ff9700 1 -- 172.21.15.194:0/1111540322 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0bbc025b30 con 0x7f0bcc0f5410 2024-01-18T06:51:17.187 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:17.188+0000 7f0bd2631700 1 -- 172.21.15.194:0/1111540322 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0bcc107000 con 0x7f0bcc0f5410 2024-01-18T06:51:17.188 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:17.189+0000 7f0bc8ff9700 1 -- 172.21.15.194:0/1111540322 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0bbc025300 con 0x7f0bcc0f5410 2024-01-18T06:51:17.188 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:51:17.188 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:51:17.190 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:17.190+0000 7f0bd2631700 1 -- 172.21.15.194:0/1111540322 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f0bb402ff60 msgr2=0x7f0bb4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:17.190 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:17.190+0000 7f0bd2631700 1 --2- 172.21.15.194:0/1111540322 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f0bb402ff60 0x7f0bb4032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f0bc00096a0 tx=0x7f0bc0006b40).stop 2024-01-18T06:51:17.190 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:17.190+0000 7f0bd2631700 1 -- 172.21.15.194:0/1111540322 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0bcc0f5410 msgr2=0x7f0bcc1055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:17.190 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:17.190+0000 7f0bd2631700 1 --2- 172.21.15.194:0/1111540322 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0bcc0f5410 0x7f0bcc1055d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f0bbc014000 tx=0x7f0bbc00b040).stop 2024-01-18T06:51:17.190 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:17.190+0000 7f0bd2631700 1 -- 172.21.15.194:0/1111540322 shutdown_connections 2024-01-18T06:51:17.190 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:17.190+0000 7f0bd2631700 1 --2- 172.21.15.194:0/1111540322 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f0bb402ff60 0x7f0bb4032420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:17.190 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:17.190+0000 7f0bd2631700 1 --2- 172.21.15.194:0/1111540322 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0bcc0f5410 0x7f0bcc1055d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:17.190 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:17.190+0000 7f0bd2631700 1 -- 172.21.15.194:0/1111540322 >> 172.21.15.194:0/1111540322 conn(0x7f0bcc0f0b70 msgr2=0x7f0bcc0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:17.190 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:17.190+0000 7f0bd2631700 1 -- 172.21.15.194:0/1111540322 shutdown_connections 2024-01-18T06:51:17.190 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:17.190+0000 7f0bd2631700 1 -- 172.21.15.194:0/1111540322 wait complete. 2024-01-18T06:51:17.191 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:51:17.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:17 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:51:15.999569+0000 mgr.smithi047.fuhzzr (mgr.14162) 84 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:51:18.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:18 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:51:17.189668+0000 mon.smithi047 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.194:0/1111540322' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:51:18.568 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:51:18.568 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:51:19.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:19 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:51:17.999924+0000 mgr.smithi047.fuhzzr (mgr.14162) 85 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:51:19.776 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:19.776+0000 7fa982fd2700 1 -- 172.21.15.194:0/1554213578 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa97c0f5410 msgr2=0x7fa97c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:19.776 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:19.776+0000 7fa982fd2700 1 --2- 172.21.15.194:0/1554213578 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa97c0f5410 0x7fa97c0f5830 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fa970009fb0 tx=0x7fa970009b30).stop 2024-01-18T06:51:19.777 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:19.776+0000 7fa982fd2700 1 -- 172.21.15.194:0/1554213578 shutdown_connections 2024-01-18T06:51:19.777 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:19.776+0000 7fa982fd2700 1 --2- 172.21.15.194:0/1554213578 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa97c0f5410 0x7fa97c0f5830 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:19.777 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:19.776+0000 7fa982fd2700 1 -- 172.21.15.194:0/1554213578 >> 172.21.15.194:0/1554213578 conn(0x7fa97c0f0b70 msgr2=0x7fa97c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:19.777 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:19.776+0000 7fa982fd2700 1 -- 172.21.15.194:0/1554213578 shutdown_connections 2024-01-18T06:51:19.777 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:19.776+0000 7fa982fd2700 1 -- 172.21.15.194:0/1554213578 wait complete. 2024-01-18T06:51:19.777 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:19.776+0000 7fa982fd2700 1 Processor -- start 2024-01-18T06:51:19.777 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:19.777+0000 7fa982fd2700 1 -- start start 2024-01-18T06:51:19.777 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:19.777+0000 7fa982fd2700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa97c0f5410 0x7fa97c1093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:19.777 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:19.777+0000 7fa982fd2700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa97c060280 con 0x7fa97c0f5410 2024-01-18T06:51:19.777 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:19.777+0000 7fa980d6e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa97c0f5410 0x7fa97c1093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:19.777 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:19.777+0000 7fa980d6e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa97c0f5410 0x7fa97c1093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:57840/0 (socket says 172.21.15.194:57840) 2024-01-18T06:51:19.777 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:19.777+0000 7fa980d6e700 1 -- 172.21.15.194:0/3931942657 learned_addr learned my addr 172.21.15.194:0/3931942657 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:51:19.779 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:19.777+0000 7fa980d6e700 1 -- 172.21.15.194:0/3931942657 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa970005040 con 0x7fa97c0f5410 2024-01-18T06:51:19.779 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:19.777+0000 7fa980d6e700 1 --2- 172.21.15.194:0/3931942657 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa97c0f5410 0x7fa97c1093f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fa970013510 tx=0x7fa97000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:19.779 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:19.778+0000 7fa9797fa700 1 -- 172.21.15.194:0/3931942657 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa970009320 con 0x7fa97c0f5410 2024-01-18T06:51:19.779 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:19.778+0000 7fa9797fa700 1 -- 172.21.15.194:0/3931942657 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa970013f00 con 0x7fa97c0f5410 2024-01-18T06:51:19.779 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:19.778+0000 7fa982fd2700 1 -- 172.21.15.194:0/3931942657 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa97c060480 con 0x7fa97c0f5410 2024-01-18T06:51:19.779 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:19.778+0000 7fa9797fa700 1 -- 172.21.15.194:0/3931942657 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa970014d90 con 0x7fa97c0f5410 2024-01-18T06:51:19.779 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:19.778+0000 7fa982fd2700 1 -- 172.21.15.194:0/3931942657 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa97c109b20 con 0x7fa97c0f5410 2024-01-18T06:51:19.779 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:19.778+0000 7fa9797fa700 1 -- 172.21.15.194:0/3931942657 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa9700095f0 con 0x7fa97c0f5410 2024-01-18T06:51:19.779 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:19.778+0000 7fa982fd2700 1 -- 172.21.15.194:0/3931942657 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa97c045ad0 con 0x7fa97c0f5410 2024-01-18T06:51:19.779 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:19.779+0000 7fa9797fa700 1 --2- 172.21.15.194:0/3931942657 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa96c02ff60 0x7fa96c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:19.780 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:19.779+0000 7fa9797fa700 1 -- 172.21.15.194:0/3931942657 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa970040fe0 con 0x7fa97c0f5410 2024-01-18T06:51:19.780 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:19.779+0000 7fa97bfff700 1 --2- 172.21.15.194:0/3931942657 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa96c02ff60 0x7fa96c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:19.780 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:19.779+0000 7fa97bfff700 1 --2- 172.21.15.194:0/3931942657 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa96c02ff60 0x7fa96c032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fa9680096a0 tx=0x7fa968006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:19.782 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:19.783+0000 7fa9797fa700 1 -- 172.21.15.194:0/3931942657 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa97001d7f0 con 0x7fa97c0f5410 2024-01-18T06:51:19.986 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:19.986+0000 7fa982fd2700 1 -- 172.21.15.194:0/3931942657 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa97c10a340 con 0x7fa97c0f5410 2024-01-18T06:51:19.986 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:19.987+0000 7fa9797fa700 1 -- 172.21.15.194:0/3931942657 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa970010230 con 0x7fa97c0f5410 2024-01-18T06:51:19.986 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:51:19.987 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:51:19.988 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:19.988+0000 7fa982fd2700 1 -- 172.21.15.194:0/3931942657 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa96c02ff60 msgr2=0x7fa96c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:19.988 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:19.988+0000 7fa982fd2700 1 --2- 172.21.15.194:0/3931942657 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa96c02ff60 0x7fa96c032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fa9680096a0 tx=0x7fa968006b40).stop 2024-01-18T06:51:19.988 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:19.988+0000 7fa982fd2700 1 -- 172.21.15.194:0/3931942657 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa97c0f5410 msgr2=0x7fa97c1093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:19.988 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:19.988+0000 7fa982fd2700 1 --2- 172.21.15.194:0/3931942657 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa97c0f5410 0x7fa97c1093f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fa970013510 tx=0x7fa97000b040).stop 2024-01-18T06:51:19.988 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:19.989+0000 7fa982fd2700 1 -- 172.21.15.194:0/3931942657 shutdown_connections 2024-01-18T06:51:19.988 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:19.989+0000 7fa982fd2700 1 --2- 172.21.15.194:0/3931942657 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa96c02ff60 0x7fa96c032420 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:19.989 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:19.989+0000 7fa982fd2700 1 --2- 172.21.15.194:0/3931942657 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa97c0f5410 0x7fa97c1093f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:19.989 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:19.989+0000 7fa982fd2700 1 -- 172.21.15.194:0/3931942657 >> 172.21.15.194:0/3931942657 conn(0x7fa97c0f0b70 msgr2=0x7fa97c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:19.989 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:19.989+0000 7fa982fd2700 1 -- 172.21.15.194:0/3931942657 shutdown_connections 2024-01-18T06:51:19.989 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:19.989+0000 7fa982fd2700 1 -- 172.21.15.194:0/3931942657 wait complete. 2024-01-18T06:51:19.989 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:51:20.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:20 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:51:19.987845+0000 mon.smithi047 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.194:0/3931942657' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:51:21.378 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:51:21.378 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:51:21.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:21 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:51:20.000181+0000 mgr.smithi047.fuhzzr (mgr.14162) 86 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:51:22.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:22.629+0000 7f2502c58700 1 -- 172.21.15.194:0/10055398 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f24fc0f62a0 msgr2=0x7f24fc0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:22.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:22.629+0000 7f2502c58700 1 --2- 172.21.15.194:0/10055398 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f24fc0f62a0 0x7f24fc0f66c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f24f8009fb0 tx=0x7f24f8009b30).stop 2024-01-18T06:51:22.631 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:22.629+0000 7f2502c58700 1 -- 172.21.15.194:0/10055398 shutdown_connections 2024-01-18T06:51:22.631 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:22.629+0000 7f2502c58700 1 --2- 172.21.15.194:0/10055398 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f24fc0f62a0 0x7f24fc0f66c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:22.631 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:22.629+0000 7f2502c58700 1 -- 172.21.15.194:0/10055398 >> 172.21.15.194:0/10055398 conn(0x7f24fc0f1aa0 msgr2=0x7f24fc0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:22.631 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:22.629+0000 7f2502c58700 1 -- 172.21.15.194:0/10055398 shutdown_connections 2024-01-18T06:51:22.631 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:22.629+0000 7f2502c58700 1 -- 172.21.15.194:0/10055398 wait complete. 2024-01-18T06:51:22.631 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:22.629+0000 7f2502c58700 1 Processor -- start 2024-01-18T06:51:22.631 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:22.629+0000 7f2502c58700 1 -- start start 2024-01-18T06:51:22.631 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:22.630+0000 7f2502c58700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f24fc0f62a0 0x7f24fc1064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:22.631 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:22.630+0000 7f2502c58700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24fc1069a0 con 0x7f24fc0f62a0 2024-01-18T06:51:22.632 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:22.630+0000 7f2501c56700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f24fc0f62a0 0x7f24fc1064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:22.632 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:22.630+0000 7f2501c56700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f24fc0f62a0 0x7f24fc1064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:57868/0 (socket says 172.21.15.194:57868) 2024-01-18T06:51:22.632 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:22.630+0000 7f2501c56700 1 -- 172.21.15.194:0/1480767442 learned_addr learned my addr 172.21.15.194:0/1480767442 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:51:22.632 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:22.630+0000 7f2501c56700 1 -- 172.21.15.194:0/1480767442 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24f8005040 con 0x7f24fc0f62a0 2024-01-18T06:51:22.632 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:22.630+0000 7f2501c56700 1 --2- 172.21.15.194:0/1480767442 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f24fc0f62a0 0x7f24fc1064b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f24f8013510 tx=0x7f24f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:22.632 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:22.630+0000 7f24f27fc700 1 -- 172.21.15.194:0/1480767442 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24f8009320 con 0x7f24fc0f62a0 2024-01-18T06:51:22.632 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:22.630+0000 7f24f27fc700 1 -- 172.21.15.194:0/1480767442 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f24f8013f00 con 0x7f24fc0f62a0 2024-01-18T06:51:22.632 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:22.630+0000 7f2502c58700 1 -- 172.21.15.194:0/1480767442 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24fc106ba0 con 0x7f24fc0f62a0 2024-01-18T06:51:22.632 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:22.630+0000 7f24f27fc700 1 -- 172.21.15.194:0/1480767442 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24f8014dc0 con 0x7f24fc0f62a0 2024-01-18T06:51:22.632 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:22.630+0000 7f2502c58700 1 -- 172.21.15.194:0/1480767442 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24fc107030 con 0x7f24fc0f62a0 2024-01-18T06:51:22.633 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:22.631+0000 7f24f27fc700 1 -- 172.21.15.194:0/1480767442 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f24f8014420 con 0x7f24fc0f62a0 2024-01-18T06:51:22.633 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:22.631+0000 7f2502c58700 1 -- 172.21.15.194:0/1480767442 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24fc045ad0 con 0x7f24fc0f62a0 2024-01-18T06:51:22.633 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:22.631+0000 7f24f27fc700 1 --2- 172.21.15.194:0/1480767442 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f24e802ffb0 0x7f24e8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:22.634 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:22.631+0000 7f24f27fc700 1 -- 172.21.15.194:0/1480767442 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f24f80410b0 con 0x7f24fc0f62a0 2024-01-18T06:51:22.634 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:22.632+0000 7f2501455700 1 --2- 172.21.15.194:0/1480767442 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f24e802ffb0 0x7f24e8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:22.634 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:22.632+0000 7f2501455700 1 --2- 172.21.15.194:0/1480767442 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f24e802ffb0 0x7f24e8032470 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f24ec0096a0 tx=0x7f24ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:22.635 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:22.636+0000 7f24f27fc700 1 -- 172.21.15.194:0/1480767442 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f24f806d2b0 con 0x7f24fc0f62a0 2024-01-18T06:51:22.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:22.840+0000 7f2502c58700 1 -- 172.21.15.194:0/1480767442 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f24fc107ee0 con 0x7f24fc0f62a0 2024-01-18T06:51:22.840 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:22.841+0000 7f24f27fc700 1 -- 172.21.15.194:0/1480767442 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f24f8026bf0 con 0x7f24fc0f62a0 2024-01-18T06:51:22.840 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:51:22.840 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:51:22.842 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:22.842+0000 7f2502c58700 1 -- 172.21.15.194:0/1480767442 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f24e802ffb0 msgr2=0x7f24e8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:22.842 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:22.842+0000 7f2502c58700 1 --2- 172.21.15.194:0/1480767442 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f24e802ffb0 0x7f24e8032470 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f24ec0096a0 tx=0x7f24ec006b40).stop 2024-01-18T06:51:22.842 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:22.842+0000 7f2502c58700 1 -- 172.21.15.194:0/1480767442 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f24fc0f62a0 msgr2=0x7f24fc1064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:22.842 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:22.842+0000 7f2502c58700 1 --2- 172.21.15.194:0/1480767442 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f24fc0f62a0 0x7f24fc1064b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f24f8013510 tx=0x7f24f800b040).stop 2024-01-18T06:51:22.842 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:22.842+0000 7f2502c58700 1 -- 172.21.15.194:0/1480767442 shutdown_connections 2024-01-18T06:51:22.842 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:22.842+0000 7f2502c58700 1 --2- 172.21.15.194:0/1480767442 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f24e802ffb0 0x7f24e8032470 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:22.842 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:22.842+0000 7f2502c58700 1 --2- 172.21.15.194:0/1480767442 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f24fc0f62a0 0x7f24fc1064b0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:22.842 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:22.842+0000 7f2502c58700 1 -- 172.21.15.194:0/1480767442 >> 172.21.15.194:0/1480767442 conn(0x7f24fc0f1aa0 msgr2=0x7f24fc0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:22.843 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:22.843+0000 7f2502c58700 1 -- 172.21.15.194:0/1480767442 shutdown_connections 2024-01-18T06:51:22.843 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:22.843+0000 7f2502c58700 1 -- 172.21.15.194:0/1480767442 wait complete. 2024-01-18T06:51:22.843 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:51:23.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:23 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:51:22.000492+0000 mgr.smithi047.fuhzzr (mgr.14162) 87 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:51:23.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:23 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:51:22.841456+0000 mon.smithi047 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.194:0/1480767442' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:51:24.241 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:51:24.241 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:51:25.434 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:25.433+0000 7f9f44fc1700 1 -- 172.21.15.194:0/1913362366 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f400f6510 msgr2=0x7f9f400f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:25.434 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:25.433+0000 7f9f44fc1700 1 --2- 172.21.15.194:0/1913362366 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f400f6510 0x7f9f400f6930 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f9f34009fb0 tx=0x7f9f34009b30).stop 2024-01-18T06:51:25.434 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:25.433+0000 7f9f44fc1700 1 -- 172.21.15.194:0/1913362366 shutdown_connections 2024-01-18T06:51:25.434 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:25.433+0000 7f9f44fc1700 1 --2- 172.21.15.194:0/1913362366 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f400f6510 0x7f9f400f6930 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:25.434 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:25.433+0000 7f9f44fc1700 1 -- 172.21.15.194:0/1913362366 >> 172.21.15.194:0/1913362366 conn(0x7f9f400f1c70 msgr2=0x7f9f400f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:25.434 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:25.433+0000 7f9f44fc1700 1 -- 172.21.15.194:0/1913362366 shutdown_connections 2024-01-18T06:51:25.434 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:25.433+0000 7f9f44fc1700 1 -- 172.21.15.194:0/1913362366 wait complete. 2024-01-18T06:51:25.436 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:25.434+0000 7f9f44fc1700 1 Processor -- start 2024-01-18T06:51:25.436 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:25.434+0000 7f9f44fc1700 1 -- start start 2024-01-18T06:51:25.436 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:25.434+0000 7f9f44fc1700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f400f6510 0x7f9f40106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:25.436 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:25.434+0000 7f9f44fc1700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f40106c20 con 0x7f9f400f6510 2024-01-18T06:51:25.436 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:25.434+0000 7f9f3e59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f400f6510 0x7f9f40106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:25.436 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:25.434+0000 7f9f3e59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f400f6510 0x7f9f40106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:57884/0 (socket says 172.21.15.194:57884) 2024-01-18T06:51:25.436 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:25.434+0000 7f9f3e59c700 1 -- 172.21.15.194:0/4103246950 learned_addr learned my addr 172.21.15.194:0/4103246950 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:51:25.436 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:25.435+0000 7f9f3e59c700 1 -- 172.21.15.194:0/4103246950 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f34005040 con 0x7f9f400f6510 2024-01-18T06:51:25.436 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:25.435+0000 7f9f3e59c700 1 --2- 172.21.15.194:0/4103246950 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f400f6510 0x7f9f40106730 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f9f34013510 tx=0x7f9f3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:25.436 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:25.435+0000 7f9f2effd700 1 -- 172.21.15.194:0/4103246950 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f34009320 con 0x7f9f400f6510 2024-01-18T06:51:25.436 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:25.435+0000 7f9f2effd700 1 -- 172.21.15.194:0/4103246950 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9f34013f00 con 0x7f9f400f6510 2024-01-18T06:51:25.436 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:25.435+0000 7f9f44fc1700 1 -- 172.21.15.194:0/4103246950 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f40106e20 con 0x7f9f400f6510 2024-01-18T06:51:25.436 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:25.435+0000 7f9f2effd700 1 -- 172.21.15.194:0/4103246950 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f34014d90 con 0x7f9f400f6510 2024-01-18T06:51:25.437 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:25.435+0000 7f9f44fc1700 1 -- 172.21.15.194:0/4103246950 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f401072b0 con 0x7f9f400f6510 2024-01-18T06:51:25.437 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:25.436+0000 7f9f2effd700 1 -- 172.21.15.194:0/4103246950 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9f34014420 con 0x7f9f400f6510 2024-01-18T06:51:25.437 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:25.436+0000 7f9f44fc1700 1 -- 172.21.15.194:0/4103246950 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f40045ad0 con 0x7f9f400f6510 2024-01-18T06:51:25.438 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:25.436+0000 7f9f2effd700 1 --2- 172.21.15.194:0/4103246950 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f9f2802ffb0 0x7f9f28032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:25.438 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:25.436+0000 7f9f2effd700 1 -- 172.21.15.194:0/4103246950 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9f340410c0 con 0x7f9f400f6510 2024-01-18T06:51:25.438 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:25.436+0000 7f9f3dd9b700 1 --2- 172.21.15.194:0/4103246950 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f9f2802ffb0 0x7f9f28032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:25.438 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:25.437+0000 7f9f3dd9b700 1 --2- 172.21.15.194:0/4103246950 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f9f2802ffb0 0x7f9f28032470 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f9f300096a0 tx=0x7f9f30006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:25.441 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:25.441+0000 7f9f2effd700 1 -- 172.21.15.194:0/4103246950 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9f340095f0 con 0x7f9f400f6510 2024-01-18T06:51:25.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:25 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:51:24.000831+0000 mgr.smithi047.fuhzzr (mgr.14162) 88 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:51:25.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:25.645+0000 7f9f44fc1700 1 -- 172.21.15.194:0/4103246950 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9f400fbf60 con 0x7f9f400f6510 2024-01-18T06:51:25.646 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:25.646+0000 7f9f2effd700 1 -- 172.21.15.194:0/4103246950 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9f3401db70 con 0x7f9f400f6510 2024-01-18T06:51:25.646 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:51:25.646 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:51:25.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:25.647+0000 7f9f44fc1700 1 -- 172.21.15.194:0/4103246950 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f9f2802ffb0 msgr2=0x7f9f28032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:25.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:25.647+0000 7f9f44fc1700 1 --2- 172.21.15.194:0/4103246950 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f9f2802ffb0 0x7f9f28032470 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f9f300096a0 tx=0x7f9f30006b40).stop 2024-01-18T06:51:25.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:25.647+0000 7f9f44fc1700 1 -- 172.21.15.194:0/4103246950 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f400f6510 msgr2=0x7f9f40106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:25.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:25.647+0000 7f9f44fc1700 1 --2- 172.21.15.194:0/4103246950 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f400f6510 0x7f9f40106730 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f9f34013510 tx=0x7f9f3400b040).stop 2024-01-18T06:51:25.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:25.647+0000 7f9f44fc1700 1 -- 172.21.15.194:0/4103246950 shutdown_connections 2024-01-18T06:51:25.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:25.647+0000 7f9f44fc1700 1 --2- 172.21.15.194:0/4103246950 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f9f2802ffb0 0x7f9f28032470 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:25.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:25.647+0000 7f9f44fc1700 1 --2- 172.21.15.194:0/4103246950 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f400f6510 0x7f9f40106730 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:25.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:25.647+0000 7f9f44fc1700 1 -- 172.21.15.194:0/4103246950 >> 172.21.15.194:0/4103246950 conn(0x7f9f400f1c70 msgr2=0x7f9f400f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:25.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:25.647+0000 7f9f44fc1700 1 -- 172.21.15.194:0/4103246950 shutdown_connections 2024-01-18T06:51:25.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:25.647+0000 7f9f44fc1700 1 -- 172.21.15.194:0/4103246950 wait complete. 2024-01-18T06:51:25.649 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:51:26.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:26 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:51:25.647581+0000 mon.smithi047 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.194:0/4103246950' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:51:27.045 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:51:27.046 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:51:27.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:27 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:51:26.001104+0000 mgr.smithi047.fuhzzr (mgr.14162) 89 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:51:28.199 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:28.196+0000 7f01903c8700 1 -- 172.21.15.194:0/167406248 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01880f5410 msgr2=0x7f01880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:28.199 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:28.196+0000 7f01903c8700 1 --2- 172.21.15.194:0/167406248 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01880f5410 0x7f01880f5830 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f0178009fb0 tx=0x7f0178009b30).stop 2024-01-18T06:51:28.199 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:28.197+0000 7f01903c8700 1 -- 172.21.15.194:0/167406248 shutdown_connections 2024-01-18T06:51:28.199 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:28.197+0000 7f01903c8700 1 --2- 172.21.15.194:0/167406248 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01880f5410 0x7f01880f5830 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:28.199 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:28.197+0000 7f01903c8700 1 -- 172.21.15.194:0/167406248 >> 172.21.15.194:0/167406248 conn(0x7f01880f0b70 msgr2=0x7f01880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:28.199 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:28.197+0000 7f01903c8700 1 -- 172.21.15.194:0/167406248 shutdown_connections 2024-01-18T06:51:28.199 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:28.197+0000 7f01903c8700 1 -- 172.21.15.194:0/167406248 wait complete. 2024-01-18T06:51:28.199 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:28.197+0000 7f01903c8700 1 Processor -- start 2024-01-18T06:51:28.199 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:28.197+0000 7f01903c8700 1 -- start start 2024-01-18T06:51:28.199 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:28.197+0000 7f01903c8700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01880f5410 0x7f018810b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:28.200 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:28.197+0000 7f01903c8700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0188107210 con 0x7f01880f5410 2024-01-18T06:51:28.200 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:28.197+0000 7f018e164700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01880f5410 0x7f018810b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:28.200 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:28.198+0000 7f018e164700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01880f5410 0x7f018810b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:53530/0 (socket says 172.21.15.194:53530) 2024-01-18T06:51:28.200 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:28.198+0000 7f018e164700 1 -- 172.21.15.194:0/862103908 learned_addr learned my addr 172.21.15.194:0/862103908 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:51:28.200 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:28.198+0000 7f018e164700 1 -- 172.21.15.194:0/862103908 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0178005040 con 0x7f01880f5410 2024-01-18T06:51:28.200 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:28.198+0000 7f018e164700 1 --2- 172.21.15.194:0/862103908 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01880f5410 0x7f018810b820 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f0178013510 tx=0x7f017800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:28.200 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:28.198+0000 7f017effd700 1 -- 172.21.15.194:0/862103908 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0178009320 con 0x7f01880f5410 2024-01-18T06:51:28.200 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:28.198+0000 7f01903c8700 1 -- 172.21.15.194:0/862103908 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0188060220 con 0x7f01880f5410 2024-01-18T06:51:28.200 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:28.198+0000 7f017effd700 1 -- 172.21.15.194:0/862103908 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0178013f00 con 0x7f01880f5410 2024-01-18T06:51:28.200 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:28.198+0000 7f01903c8700 1 -- 172.21.15.194:0/862103908 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f018810bd10 con 0x7f01880f5410 2024-01-18T06:51:28.200 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:28.198+0000 7f017effd700 1 -- 172.21.15.194:0/862103908 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0178014d90 con 0x7f01880f5410 2024-01-18T06:51:28.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:28.199+0000 7f017effd700 1 -- 172.21.15.194:0/862103908 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f01780095f0 con 0x7f01880f5410 2024-01-18T06:51:28.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:28.199+0000 7f01903c8700 1 -- 172.21.15.194:0/862103908 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0188045ad0 con 0x7f01880f5410 2024-01-18T06:51:28.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:28.199+0000 7f017effd700 1 --2- 172.21.15.194:0/862103908 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f017402ff60 0x7f0174032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:28.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:28.199+0000 7f017effd700 1 -- 172.21.15.194:0/862103908 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0178041090 con 0x7f01880f5410 2024-01-18T06:51:28.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:28.199+0000 7f018d963700 1 --2- 172.21.15.194:0/862103908 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f017402ff60 0x7f0174032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:28.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:28.200+0000 7f018d963700 1 --2- 172.21.15.194:0/862103908 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f017402ff60 0x7f0174032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f01840096a0 tx=0x7f0184006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:28.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:28.204+0000 7f017effd700 1 -- 172.21.15.194:0/862103908 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0178014420 con 0x7f01880f5410 2024-01-18T06:51:28.411 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:28.410+0000 7f01903c8700 1 -- 172.21.15.194:0/862103908 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f01880fae60 con 0x7f01880f5410 2024-01-18T06:51:28.412 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:28.411+0000 7f017effd700 1 -- 172.21.15.194:0/862103908 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0178010230 con 0x7f01880f5410 2024-01-18T06:51:28.412 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:51:28.412 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:51:28.413 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:28.413+0000 7f01903c8700 1 -- 172.21.15.194:0/862103908 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f017402ff60 msgr2=0x7f0174032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:28.413 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:28.413+0000 7f01903c8700 1 --2- 172.21.15.194:0/862103908 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f017402ff60 0x7f0174032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f01840096a0 tx=0x7f0184006b40).stop 2024-01-18T06:51:28.413 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:28.413+0000 7f01903c8700 1 -- 172.21.15.194:0/862103908 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01880f5410 msgr2=0x7f018810b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:28.413 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:28.413+0000 7f01903c8700 1 --2- 172.21.15.194:0/862103908 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01880f5410 0x7f018810b820 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f0178013510 tx=0x7f017800b040).stop 2024-01-18T06:51:28.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:28.413+0000 7f01903c8700 1 -- 172.21.15.194:0/862103908 shutdown_connections 2024-01-18T06:51:28.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:28.413+0000 7f01903c8700 1 --2- 172.21.15.194:0/862103908 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f017402ff60 0x7f0174032420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:28.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:28.413+0000 7f01903c8700 1 --2- 172.21.15.194:0/862103908 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01880f5410 0x7f018810b820 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:28.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:28.413+0000 7f01903c8700 1 -- 172.21.15.194:0/862103908 >> 172.21.15.194:0/862103908 conn(0x7f01880f0b70 msgr2=0x7f01880f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:28.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:28.413+0000 7f01903c8700 1 -- 172.21.15.194:0/862103908 shutdown_connections 2024-01-18T06:51:28.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:28.413+0000 7f01903c8700 1 -- 172.21.15.194:0/862103908 wait complete. 2024-01-18T06:51:28.414 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:51:29.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:28 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:51:28.001426+0000 mgr.smithi047.fuhzzr (mgr.14162) 90 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:51:29.214 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:28 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:51:28.412963+0000 mon.smithi047 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.194:0/862103908' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:51:29.822 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:51:29.822 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:51:31.011 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:31.010+0000 7fa3e151c700 1 -- 172.21.15.194:0/3136648557 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3dc0f47b0 msgr2=0x7fa3dc0f4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:31.012 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:31.010+0000 7fa3e151c700 1 --2- 172.21.15.194:0/3136648557 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3dc0f47b0 0x7fa3dc0f4bd0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fa3d0009fb0 tx=0x7fa3d0009b30).stop 2024-01-18T06:51:31.012 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:31.010+0000 7fa3e151c700 1 -- 172.21.15.194:0/3136648557 shutdown_connections 2024-01-18T06:51:31.012 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:31.010+0000 7fa3e151c700 1 --2- 172.21.15.194:0/3136648557 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3dc0f47b0 0x7fa3dc0f4bd0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:31.012 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:31.010+0000 7fa3e151c700 1 -- 172.21.15.194:0/3136648557 >> 172.21.15.194:0/3136648557 conn(0x7fa3dc0f02b0 msgr2=0x7fa3dc0f26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:31.012 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:31.010+0000 7fa3e151c700 1 -- 172.21.15.194:0/3136648557 shutdown_connections 2024-01-18T06:51:31.012 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:31.010+0000 7fa3e151c700 1 -- 172.21.15.194:0/3136648557 wait complete. 2024-01-18T06:51:31.012 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:31.011+0000 7fa3e151c700 1 Processor -- start 2024-01-18T06:51:31.012 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:31.011+0000 7fa3e151c700 1 -- start start 2024-01-18T06:51:31.012 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:31.011+0000 7fa3e151c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3dc0f47b0 0x7fa3dc1066a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:31.012 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:31.011+0000 7fa3e151c700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3dc106b90 con 0x7fa3dc0f47b0 2024-01-18T06:51:31.015 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:31.011+0000 7fa3daffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3dc0f47b0 0x7fa3dc1066a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:31.015 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:31.011+0000 7fa3daffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3dc0f47b0 0x7fa3dc1066a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:53536/0 (socket says 172.21.15.194:53536) 2024-01-18T06:51:31.016 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:31.011+0000 7fa3daffd700 1 -- 172.21.15.194:0/4275509200 learned_addr learned my addr 172.21.15.194:0/4275509200 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:51:31.016 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:31.012+0000 7fa3daffd700 1 -- 172.21.15.194:0/4275509200 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d0005040 con 0x7fa3dc0f47b0 2024-01-18T06:51:31.016 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:31.012+0000 7fa3daffd700 1 --2- 172.21.15.194:0/4275509200 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3dc0f47b0 0x7fa3dc1066a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fa3d0013510 tx=0x7fa3d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:31.016 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:31.012+0000 7fa3c37fe700 1 -- 172.21.15.194:0/4275509200 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3d0009320 con 0x7fa3dc0f47b0 2024-01-18T06:51:31.016 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:31.012+0000 7fa3e151c700 1 -- 172.21.15.194:0/4275509200 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3dc106d90 con 0x7fa3dc0f47b0 2024-01-18T06:51:31.017 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:31.012+0000 7fa3c37fe700 1 -- 172.21.15.194:0/4275509200 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa3d0013f00 con 0x7fa3dc0f47b0 2024-01-18T06:51:31.017 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:31.012+0000 7fa3c37fe700 1 -- 172.21.15.194:0/4275509200 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3d0014d90 con 0x7fa3dc0f47b0 2024-01-18T06:51:31.017 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:31.012+0000 7fa3e151c700 1 -- 172.21.15.194:0/4275509200 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3dc107220 con 0x7fa3dc0f47b0 2024-01-18T06:51:31.017 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:31.013+0000 7fa3c37fe700 1 -- 172.21.15.194:0/4275509200 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa3d00095f0 con 0x7fa3dc0f47b0 2024-01-18T06:51:31.017 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:31.013+0000 7fa3e151c700 1 -- 172.21.15.194:0/4275509200 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3dc045ad0 con 0x7fa3dc0f47b0 2024-01-18T06:51:31.017 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:31.013+0000 7fa3c37fe700 1 --2- 172.21.15.194:0/4275509200 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa3c402ff60 0x7fa3c4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:31.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:31.013+0000 7fa3c37fe700 1 -- 172.21.15.194:0/4275509200 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa3d0040fc0 con 0x7fa3dc0f47b0 2024-01-18T06:51:31.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:31.013+0000 7fa3da7fc700 1 --2- 172.21.15.194:0/4275509200 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa3c402ff60 0x7fa3c4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:31.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:31.014+0000 7fa3da7fc700 1 --2- 172.21.15.194:0/4275509200 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa3c402ff60 0x7fa3c4032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fa3cc0096a0 tx=0x7fa3cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:31.019 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:31.018+0000 7fa3c37fe700 1 -- 172.21.15.194:0/4275509200 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa3d0014420 con 0x7fa3dc0f47b0 2024-01-18T06:51:31.228 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:31.227+0000 7fa3e151c700 1 -- 172.21.15.194:0/4275509200 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa3dc108070 con 0x7fa3dc0f47b0 2024-01-18T06:51:31.228 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:31.227+0000 7fa3c37fe700 1 -- 172.21.15.194:0/4275509200 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa3d0010230 con 0x7fa3dc0f47b0 2024-01-18T06:51:31.228 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:51:31.229 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:51:31.230 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:31.229+0000 7fa3e151c700 1 -- 172.21.15.194:0/4275509200 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa3c402ff60 msgr2=0x7fa3c4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:31.230 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:31.229+0000 7fa3e151c700 1 --2- 172.21.15.194:0/4275509200 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa3c402ff60 0x7fa3c4032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fa3cc0096a0 tx=0x7fa3cc006b40).stop 2024-01-18T06:51:31.230 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:31.229+0000 7fa3e151c700 1 -- 172.21.15.194:0/4275509200 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3dc0f47b0 msgr2=0x7fa3dc1066a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:31.230 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:31.229+0000 7fa3e151c700 1 --2- 172.21.15.194:0/4275509200 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3dc0f47b0 0x7fa3dc1066a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fa3d0013510 tx=0x7fa3d000b040).stop 2024-01-18T06:51:31.230 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:31.229+0000 7fa3e151c700 1 -- 172.21.15.194:0/4275509200 shutdown_connections 2024-01-18T06:51:31.230 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:31.229+0000 7fa3e151c700 1 --2- 172.21.15.194:0/4275509200 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa3c402ff60 0x7fa3c4032420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:31.230 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:31.229+0000 7fa3e151c700 1 --2- 172.21.15.194:0/4275509200 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa3dc0f47b0 0x7fa3dc1066a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:31.230 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:31.229+0000 7fa3e151c700 1 -- 172.21.15.194:0/4275509200 >> 172.21.15.194:0/4275509200 conn(0x7fa3dc0f02b0 msgr2=0x7fa3dc0f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:31.230 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:31.229+0000 7fa3e151c700 1 -- 172.21.15.194:0/4275509200 shutdown_connections 2024-01-18T06:51:31.230 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:31.229+0000 7fa3e151c700 1 -- 172.21.15.194:0/4275509200 wait complete. 2024-01-18T06:51:31.231 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:51:31.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:31 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:51:30.001751+0000 mgr.smithi047.fuhzzr (mgr.14162) 91 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:51:31.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:31 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:51:30.113295+0000 mon.smithi047 (mon.0) 174 : audit [INF] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.fuhzzr/mirror_snapshot_schedule"}]: dispatch 2024-01-18T06:51:31.464 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:31 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:51:30.115430+0000 mon.smithi047 (mon.0) 175 : audit [INF] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.fuhzzr/trash_purge_schedule"}]: dispatch 2024-01-18T06:51:32.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:32 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:51:31.229471+0000 mon.smithi047 (mon.0) 176 : audit [DBG] from='client.? 172.21.15.194:0/4275509200' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:51:32.633 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:51:32.634 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:51:33.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:33 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:51:32.002089+0000 mgr.smithi047.fuhzzr (mgr.14162) 92 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:51:33.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:33.878+0000 7f555e078700 1 -- 172.21.15.194:0/1691797331 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f55580f5410 msgr2=0x7f55580f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:33.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:33.878+0000 7f555e078700 1 --2- 172.21.15.194:0/1691797331 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f55580f5410 0x7f55580f5830 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f554c009fb0 tx=0x7f554c009b30).stop 2024-01-18T06:51:33.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:33.878+0000 7f555e078700 1 -- 172.21.15.194:0/1691797331 shutdown_connections 2024-01-18T06:51:33.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:33.878+0000 7f555e078700 1 --2- 172.21.15.194:0/1691797331 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f55580f5410 0x7f55580f5830 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:33.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:33.878+0000 7f555e078700 1 -- 172.21.15.194:0/1691797331 >> 172.21.15.194:0/1691797331 conn(0x7f55580f0b70 msgr2=0x7f55580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:33.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:33.879+0000 7f555e078700 1 -- 172.21.15.194:0/1691797331 shutdown_connections 2024-01-18T06:51:33.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:33.879+0000 7f555e078700 1 -- 172.21.15.194:0/1691797331 wait complete. 2024-01-18T06:51:33.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:33.879+0000 7f555e078700 1 Processor -- start 2024-01-18T06:51:33.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:33.879+0000 7f555e078700 1 -- start start 2024-01-18T06:51:33.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:33.879+0000 7f555e078700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f55580f5410 0x7f55581055d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:33.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:33.879+0000 7f555e078700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5558105ac0 con 0x7f55580f5410 2024-01-18T06:51:33.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:33.880+0000 7f55577fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f55580f5410 0x7f55581055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:33.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:33.880+0000 7f55577fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f55580f5410 0x7f55581055d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:53554/0 (socket says 172.21.15.194:53554) 2024-01-18T06:51:33.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:33.880+0000 7f55577fe700 1 -- 172.21.15.194:0/1767162275 learned_addr learned my addr 172.21.15.194:0/1767162275 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:51:33.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:33.880+0000 7f55577fe700 1 -- 172.21.15.194:0/1767162275 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f554c005040 con 0x7f55580f5410 2024-01-18T06:51:33.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:33.880+0000 7f55577fe700 1 --2- 172.21.15.194:0/1767162275 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f55580f5410 0x7f55581055d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f554c013510 tx=0x7f554c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:33.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:33.880+0000 7f553ffff700 1 -- 172.21.15.194:0/1767162275 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f554c009320 con 0x7f55580f5410 2024-01-18T06:51:33.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:33.880+0000 7f555e078700 1 -- 172.21.15.194:0/1767162275 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5558105cc0 con 0x7f55580f5410 2024-01-18T06:51:33.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:33.880+0000 7f553ffff700 1 -- 172.21.15.194:0/1767162275 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f554c013f00 con 0x7f55580f5410 2024-01-18T06:51:33.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:33.880+0000 7f553ffff700 1 -- 172.21.15.194:0/1767162275 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f554c014d90 con 0x7f55580f5410 2024-01-18T06:51:33.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:33.880+0000 7f555e078700 1 -- 172.21.15.194:0/1767162275 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5558106150 con 0x7f55580f5410 2024-01-18T06:51:33.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:33.881+0000 7f553ffff700 1 -- 172.21.15.194:0/1767162275 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f554c014420 con 0x7f55580f5410 2024-01-18T06:51:33.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:33.881+0000 7f555e078700 1 -- 172.21.15.194:0/1767162275 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5558045ad0 con 0x7f55580f5410 2024-01-18T06:51:33.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:33.881+0000 7f553ffff700 1 --2- 172.21.15.194:0/1767162275 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f554002ff60 0x7f5540032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:33.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:33.881+0000 7f553ffff700 1 -- 172.21.15.194:0/1767162275 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f554c0410c0 con 0x7f55580f5410 2024-01-18T06:51:33.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:33.881+0000 7f5556ffd700 1 --2- 172.21.15.194:0/1767162275 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f554002ff60 0x7f5540032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:33.884 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:33.882+0000 7f5556ffd700 1 --2- 172.21.15.194:0/1767162275 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f554002ff60 0x7f5540032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f55480096a0 tx=0x7f5548006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:33.886 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:33.887+0000 7f553ffff700 1 -- 172.21.15.194:0/1767162275 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f554c0095f0 con 0x7f55580f5410 2024-01-18T06:51:34.091 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:34.090+0000 7f555e078700 1 -- 172.21.15.194:0/1767162275 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5558107000 con 0x7f55580f5410 2024-01-18T06:51:34.091 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:34.091+0000 7f553ffff700 1 -- 172.21.15.194:0/1767162275 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f554c01db70 con 0x7f55580f5410 2024-01-18T06:51:34.092 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:51:34.092 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:51:34.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:34.092+0000 7f555e078700 1 -- 172.21.15.194:0/1767162275 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f554002ff60 msgr2=0x7f5540032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:34.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:34.092+0000 7f555e078700 1 --2- 172.21.15.194:0/1767162275 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f554002ff60 0x7f5540032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f55480096a0 tx=0x7f5548006b40).stop 2024-01-18T06:51:34.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:34.092+0000 7f555e078700 1 -- 172.21.15.194:0/1767162275 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f55580f5410 msgr2=0x7f55581055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:34.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:34.092+0000 7f555e078700 1 --2- 172.21.15.194:0/1767162275 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f55580f5410 0x7f55581055d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f554c013510 tx=0x7f554c00b040).stop 2024-01-18T06:51:34.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:34.093+0000 7f555e078700 1 -- 172.21.15.194:0/1767162275 shutdown_connections 2024-01-18T06:51:34.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:34.093+0000 7f555e078700 1 --2- 172.21.15.194:0/1767162275 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f554002ff60 0x7f5540032420 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:34.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:34.093+0000 7f555e078700 1 --2- 172.21.15.194:0/1767162275 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f55580f5410 0x7f55581055d0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:34.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:34.093+0000 7f555e078700 1 -- 172.21.15.194:0/1767162275 >> 172.21.15.194:0/1767162275 conn(0x7f55580f0b70 msgr2=0x7f55580f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:34.094 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:34.093+0000 7f555e078700 1 -- 172.21.15.194:0/1767162275 shutdown_connections 2024-01-18T06:51:34.094 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:34.093+0000 7f555e078700 1 -- 172.21.15.194:0/1767162275 wait complete. 2024-01-18T06:51:34.094 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:51:35.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:35 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:51:34.002325+0000 mgr.smithi047.fuhzzr (mgr.14162) 93 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:51:35.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:35 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:51:34.092946+0000 mon.smithi047 (mon.0) 177 : audit [DBG] from='client.? 172.21.15.194:0/1767162275' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:51:35.485 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:51:35.485 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:51:36.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:36.727+0000 7f0b09aef700 1 -- 172.21.15.194:0/2071155223 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b04060280 msgr2=0x7f0b040f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:36.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:36.727+0000 7f0b09aef700 1 --2- 172.21.15.194:0/2071155223 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b04060280 0x7f0b040f90f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f0aec009fb0 tx=0x7f0aec009b30).stop 2024-01-18T06:51:36.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:36.727+0000 7f0b09aef700 1 -- 172.21.15.194:0/2071155223 shutdown_connections 2024-01-18T06:51:36.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:36.727+0000 7f0b09aef700 1 --2- 172.21.15.194:0/2071155223 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b04060280 0x7f0b040f90f0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:36.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:36.727+0000 7f0b09aef700 1 -- 172.21.15.194:0/2071155223 >> 172.21.15.194:0/2071155223 conn(0x7f0b040f0b70 msgr2=0x7f0b040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:36.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:36.727+0000 7f0b09aef700 1 -- 172.21.15.194:0/2071155223 shutdown_connections 2024-01-18T06:51:36.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:36.727+0000 7f0b09aef700 1 -- 172.21.15.194:0/2071155223 wait complete. 2024-01-18T06:51:36.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:36.728+0000 7f0b09aef700 1 Processor -- start 2024-01-18T06:51:36.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:36.728+0000 7f0b09aef700 1 -- start start 2024-01-18T06:51:36.729 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:36.728+0000 7f0b09aef700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b04060280 0x7f0b040f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:36.729 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:36.728+0000 7f0b09aef700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b040f4ff0 con 0x7f0b04060280 2024-01-18T06:51:36.729 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:36.728+0000 7f0b037fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b04060280 0x7f0b040f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:36.729 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:36.728+0000 7f0b037fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b04060280 0x7f0b040f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:53566/0 (socket says 172.21.15.194:53566) 2024-01-18T06:51:36.729 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:36.728+0000 7f0b037fe700 1 -- 172.21.15.194:0/74705960 learned_addr learned my addr 172.21.15.194:0/74705960 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:51:36.729 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:36.728+0000 7f0b037fe700 1 -- 172.21.15.194:0/74705960 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aec005040 con 0x7f0b04060280 2024-01-18T06:51:36.729 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:36.729+0000 7f0b037fe700 1 --2- 172.21.15.194:0/74705960 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b04060280 0x7f0b040f6b70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f0aec013510 tx=0x7f0aec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:36.730 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:36.729+0000 7f0afbfff700 1 -- 172.21.15.194:0/74705960 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0aec009320 con 0x7f0b04060280 2024-01-18T06:51:36.730 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:36.729+0000 7f0b09aef700 1 -- 172.21.15.194:0/74705960 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b040f5190 con 0x7f0b04060280 2024-01-18T06:51:36.730 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:36.729+0000 7f0afbfff700 1 -- 172.21.15.194:0/74705960 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0aec013f00 con 0x7f0b04060280 2024-01-18T06:51:36.730 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:36.729+0000 7f0afbfff700 1 -- 172.21.15.194:0/74705960 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0aec014dc0 con 0x7f0b04060280 2024-01-18T06:51:36.730 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:36.729+0000 7f0b09aef700 1 -- 172.21.15.194:0/74705960 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b040f5620 con 0x7f0b04060280 2024-01-18T06:51:36.731 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:36.730+0000 7f0afbfff700 1 -- 172.21.15.194:0/74705960 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0aec0095f0 con 0x7f0b04060280 2024-01-18T06:51:36.731 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:36.730+0000 7f0b09aef700 1 -- 172.21.15.194:0/74705960 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b04059100 con 0x7f0b04060280 2024-01-18T06:51:36.731 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:36.730+0000 7f0afbfff700 1 --2- 172.21.15.194:0/74705960 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f0af002ff60 0x7f0af0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:36.731 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:36.730+0000 7f0afbfff700 1 -- 172.21.15.194:0/74705960 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0aec041020 con 0x7f0b04060280 2024-01-18T06:51:36.731 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:36.730+0000 7f0b02ffd700 1 --2- 172.21.15.194:0/74705960 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f0af002ff60 0x7f0af0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:36.731 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:36.730+0000 7f0b02ffd700 1 --2- 172.21.15.194:0/74705960 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f0af002ff60 0x7f0af0032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f0af40096a0 tx=0x7f0af4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:36.734 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:36.734+0000 7f0afbfff700 1 -- 172.21.15.194:0/74705960 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0aec014420 con 0x7f0b04060280 2024-01-18T06:51:36.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:36.940+0000 7f0b09aef700 1 -- 172.21.15.194:0/74705960 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0b04045ad0 con 0x7f0b04060280 2024-01-18T06:51:36.940 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:36.941+0000 7f0afbfff700 1 -- 172.21.15.194:0/74705960 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0aec01c920 con 0x7f0b04060280 2024-01-18T06:51:36.940 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:51:36.940 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:51:36.942 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:36.942+0000 7f0b09aef700 1 -- 172.21.15.194:0/74705960 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f0af002ff60 msgr2=0x7f0af0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:36.942 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:36.942+0000 7f0b09aef700 1 --2- 172.21.15.194:0/74705960 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f0af002ff60 0x7f0af0032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f0af40096a0 tx=0x7f0af4006b40).stop 2024-01-18T06:51:36.942 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:36.942+0000 7f0b09aef700 1 -- 172.21.15.194:0/74705960 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b04060280 msgr2=0x7f0b040f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:36.942 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:36.942+0000 7f0b09aef700 1 --2- 172.21.15.194:0/74705960 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b04060280 0x7f0b040f6b70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f0aec013510 tx=0x7f0aec00b040).stop 2024-01-18T06:51:36.942 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:36.942+0000 7f0b09aef700 1 -- 172.21.15.194:0/74705960 shutdown_connections 2024-01-18T06:51:36.942 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:36.942+0000 7f0b09aef700 1 --2- 172.21.15.194:0/74705960 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f0af002ff60 0x7f0af0032420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:36.942 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:36.942+0000 7f0b09aef700 1 --2- 172.21.15.194:0/74705960 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b04060280 0x7f0b040f6b70 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:36.943 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:36.942+0000 7f0b09aef700 1 -- 172.21.15.194:0/74705960 >> 172.21.15.194:0/74705960 conn(0x7f0b040f0b70 msgr2=0x7f0b040f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:36.943 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:36.942+0000 7f0b09aef700 1 -- 172.21.15.194:0/74705960 shutdown_connections 2024-01-18T06:51:36.943 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:36.942+0000 7f0b09aef700 1 -- 172.21.15.194:0/74705960 wait complete. 2024-01-18T06:51:36.943 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:51:37.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:51:36.002595+0000 mgr.smithi047.fuhzzr (mgr.14162) 94 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:51:37.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:51:36.941312+0000 mon.smithi047 (mon.0) 178 : audit [DBG] from='client.? 172.21.15.194:0/74705960' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:51:38.334 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:51:38.334 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:51:39.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:38 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:51:38.002847+0000 mgr.smithi047.fuhzzr (mgr.14162) 95 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:51:39.593 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:39.592+0000 7fd238f27700 1 -- 172.21.15.194:0/3084734469 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2340f6510 msgr2=0x7fd2340f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:39.594 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:39.592+0000 7fd238f27700 1 --2- 172.21.15.194:0/3084734469 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2340f6510 0x7fd2340f6930 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7fd224009fb0 tx=0x7fd224009b30).stop 2024-01-18T06:51:39.594 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:39.592+0000 7fd238f27700 1 -- 172.21.15.194:0/3084734469 shutdown_connections 2024-01-18T06:51:39.594 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:39.592+0000 7fd238f27700 1 --2- 172.21.15.194:0/3084734469 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2340f6510 0x7fd2340f6930 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:39.594 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:39.592+0000 7fd238f27700 1 -- 172.21.15.194:0/3084734469 >> 172.21.15.194:0/3084734469 conn(0x7fd2340f1c70 msgr2=0x7fd2340f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:39.594 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:39.592+0000 7fd238f27700 1 -- 172.21.15.194:0/3084734469 shutdown_connections 2024-01-18T06:51:39.594 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:39.592+0000 7fd238f27700 1 -- 172.21.15.194:0/3084734469 wait complete. 2024-01-18T06:51:39.594 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:39.593+0000 7fd238f27700 1 Processor -- start 2024-01-18T06:51:39.594 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:39.593+0000 7fd238f27700 1 -- start start 2024-01-18T06:51:39.595 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:39.593+0000 7fd238f27700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2340f6510 0x7fd234106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:39.595 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:39.593+0000 7fd238f27700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd234106c20 con 0x7fd2340f6510 2024-01-18T06:51:39.595 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:39.593+0000 7fd23259c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2340f6510 0x7fd234106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:39.595 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:39.593+0000 7fd23259c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2340f6510 0x7fd234106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:51444/0 (socket says 172.21.15.194:51444) 2024-01-18T06:51:39.595 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:39.593+0000 7fd23259c700 1 -- 172.21.15.194:0/2293628016 learned_addr learned my addr 172.21.15.194:0/2293628016 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:51:39.595 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:39.594+0000 7fd23259c700 1 -- 172.21.15.194:0/2293628016 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd224005040 con 0x7fd2340f6510 2024-01-18T06:51:39.595 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:39.594+0000 7fd23259c700 1 --2- 172.21.15.194:0/2293628016 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2340f6510 0x7fd234106730 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fd224014000 tx=0x7fd22400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:39.595 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:39.594+0000 7fd222ffd700 1 -- 172.21.15.194:0/2293628016 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd2240095f0 con 0x7fd2340f6510 2024-01-18T06:51:39.595 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:39.594+0000 7fd222ffd700 1 -- 172.21.15.194:0/2293628016 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd224007e70 con 0x7fd2340f6510 2024-01-18T06:51:39.595 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:39.594+0000 7fd238f27700 1 -- 172.21.15.194:0/2293628016 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd234106e20 con 0x7fd2340f6510 2024-01-18T06:51:39.595 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:39.594+0000 7fd222ffd700 1 -- 172.21.15.194:0/2293628016 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd224014ea0 con 0x7fd2340f6510 2024-01-18T06:51:39.595 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:39.594+0000 7fd238f27700 1 -- 172.21.15.194:0/2293628016 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2341072b0 con 0x7fd2340f6510 2024-01-18T06:51:39.596 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:39.595+0000 7fd222ffd700 1 -- 172.21.15.194:0/2293628016 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd224025070 con 0x7fd2340f6510 2024-01-18T06:51:39.597 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:39.595+0000 7fd238f27700 1 -- 172.21.15.194:0/2293628016 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd234045ad0 con 0x7fd2340f6510 2024-01-18T06:51:39.597 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:39.595+0000 7fd222ffd700 1 --2- 172.21.15.194:0/2293628016 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fd21c02ff60 0x7fd21c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:39.597 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:39.595+0000 7fd222ffd700 1 -- 172.21.15.194:0/2293628016 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd224040980 con 0x7fd2340f6510 2024-01-18T06:51:39.597 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:39.595+0000 7fd231d9b700 1 --2- 172.21.15.194:0/2293628016 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fd21c02ff60 0x7fd21c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:39.597 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:39.596+0000 7fd231d9b700 1 --2- 172.21.15.194:0/2293628016 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fd21c02ff60 0x7fd21c032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fd2280096a0 tx=0x7fd228006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:39.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:39.600+0000 7fd222ffd700 1 -- 172.21.15.194:0/2293628016 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd2240079c0 con 0x7fd2340f6510 2024-01-18T06:51:39.803 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:39.804+0000 7fd238f27700 1 -- 172.21.15.194:0/2293628016 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd2340fbf60 con 0x7fd2340f6510 2024-01-18T06:51:39.804 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:39.804+0000 7fd222ffd700 1 -- 172.21.15.194:0/2293628016 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd22400f300 con 0x7fd2340f6510 2024-01-18T06:51:39.804 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:51:39.804 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:51:39.805 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:39.806+0000 7fd238f27700 1 -- 172.21.15.194:0/2293628016 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fd21c02ff60 msgr2=0x7fd21c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:39.805 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:39.806+0000 7fd238f27700 1 --2- 172.21.15.194:0/2293628016 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fd21c02ff60 0x7fd21c032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fd2280096a0 tx=0x7fd228006b40).stop 2024-01-18T06:51:39.806 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:39.806+0000 7fd238f27700 1 -- 172.21.15.194:0/2293628016 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2340f6510 msgr2=0x7fd234106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:39.806 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:39.806+0000 7fd238f27700 1 --2- 172.21.15.194:0/2293628016 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2340f6510 0x7fd234106730 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fd224014000 tx=0x7fd22400b040).stop 2024-01-18T06:51:39.806 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:39.806+0000 7fd238f27700 1 -- 172.21.15.194:0/2293628016 shutdown_connections 2024-01-18T06:51:39.806 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:39.806+0000 7fd238f27700 1 --2- 172.21.15.194:0/2293628016 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fd21c02ff60 0x7fd21c032420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:39.806 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:39.806+0000 7fd238f27700 1 --2- 172.21.15.194:0/2293628016 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2340f6510 0x7fd234106730 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:39.806 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:39.806+0000 7fd238f27700 1 -- 172.21.15.194:0/2293628016 >> 172.21.15.194:0/2293628016 conn(0x7fd2340f1c70 msgr2=0x7fd2340f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:39.806 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:39.806+0000 7fd238f27700 1 -- 172.21.15.194:0/2293628016 shutdown_connections 2024-01-18T06:51:39.806 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:39.806+0000 7fd238f27700 1 -- 172.21.15.194:0/2293628016 wait complete. 2024-01-18T06:51:39.806 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:51:40.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:39 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:51:39.805185+0000 mon.smithi047 (mon.0) 179 : audit [DBG] from='client.? 172.21.15.194:0/2293628016' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:51:41.189 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:51:41.226 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:51:41.227 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:40 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:51:40.003192+0000 mgr.smithi047.fuhzzr (mgr.14162) 96 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:51:42.450 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:42.451+0000 7febf67e6700 1 -- 172.21.15.194:0/2750461653 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7febf00f62a0 msgr2=0x7febf00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:42.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:42.451+0000 7febf67e6700 1 --2- 172.21.15.194:0/2750461653 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7febf00f62a0 0x7febf00f66c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7febe0009fb0 tx=0x7febe0009b30).stop 2024-01-18T06:51:42.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:42.451+0000 7febf67e6700 1 -- 172.21.15.194:0/2750461653 shutdown_connections 2024-01-18T06:51:42.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:42.451+0000 7febf67e6700 1 --2- 172.21.15.194:0/2750461653 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7febf00f62a0 0x7febf00f66c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:42.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:42.451+0000 7febf67e6700 1 -- 172.21.15.194:0/2750461653 >> 172.21.15.194:0/2750461653 conn(0x7febf00f1aa0 msgr2=0x7febf00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:42.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:42.451+0000 7febf67e6700 1 -- 172.21.15.194:0/2750461653 shutdown_connections 2024-01-18T06:51:42.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:42.451+0000 7febf67e6700 1 -- 172.21.15.194:0/2750461653 wait complete. 2024-01-18T06:51:42.452 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:42.451+0000 7febf67e6700 1 Processor -- start 2024-01-18T06:51:42.453 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:42.451+0000 7febf67e6700 1 -- start start 2024-01-18T06:51:42.453 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:42.452+0000 7febf67e6700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7febf00f62a0 0x7febf01064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:42.453 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:42.452+0000 7febf67e6700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf01069a0 con 0x7febf00f62a0 2024-01-18T06:51:42.453 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:42.452+0000 7febf57e4700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7febf00f62a0 0x7febf01064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:42.453 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:42.452+0000 7febf57e4700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7febf00f62a0 0x7febf01064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:51472/0 (socket says 172.21.15.194:51472) 2024-01-18T06:51:42.453 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:42.452+0000 7febf57e4700 1 -- 172.21.15.194:0/601971833 learned_addr learned my addr 172.21.15.194:0/601971833 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:51:42.453 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:42.452+0000 7febf57e4700 1 -- 172.21.15.194:0/601971833 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe0005040 con 0x7febf00f62a0 2024-01-18T06:51:42.453 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:42.452+0000 7febf57e4700 1 --2- 172.21.15.194:0/601971833 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7febf00f62a0 0x7febf01064b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7febe0013510 tx=0x7febe000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:42.453 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:42.452+0000 7febe5ffb700 1 -- 172.21.15.194:0/601971833 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febe0009320 con 0x7febf00f62a0 2024-01-18T06:51:42.453 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:42.453+0000 7febe5ffb700 1 -- 172.21.15.194:0/601971833 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7febe0013f00 con 0x7febf00f62a0 2024-01-18T06:51:42.453 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:42.453+0000 7febe5ffb700 1 -- 172.21.15.194:0/601971833 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febe0014dc0 con 0x7febf00f62a0 2024-01-18T06:51:42.453 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:42.453+0000 7febf67e6700 1 -- 172.21.15.194:0/601971833 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febf0106ba0 con 0x7febf00f62a0 2024-01-18T06:51:42.453 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:42.453+0000 7febf67e6700 1 -- 172.21.15.194:0/601971833 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febf0107030 con 0x7febf00f62a0 2024-01-18T06:51:42.454 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:42.454+0000 7febe5ffb700 1 -- 172.21.15.194:0/601971833 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7febe00095f0 con 0x7febf00f62a0 2024-01-18T06:51:42.454 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:42.454+0000 7febf67e6700 1 -- 172.21.15.194:0/601971833 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febf0045ad0 con 0x7febf00f62a0 2024-01-18T06:51:42.454 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:42.454+0000 7febe5ffb700 1 --2- 172.21.15.194:0/601971833 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7febdc02ff60 0x7febdc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:42.454 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:42.454+0000 7febe5ffb700 1 -- 172.21.15.194:0/601971833 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7febe00410a0 con 0x7febf00f62a0 2024-01-18T06:51:42.454 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:42.454+0000 7febf4fe3700 1 --2- 172.21.15.194:0/601971833 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7febdc02ff60 0x7febdc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:42.454 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:42.455+0000 7febf4fe3700 1 --2- 172.21.15.194:0/601971833 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7febdc02ff60 0x7febdc032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7febec0096a0 tx=0x7febec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:42.458 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:42.459+0000 7febe5ffb700 1 -- 172.21.15.194:0/601971833 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7febe0014420 con 0x7febf00f62a0 2024-01-18T06:51:42.663 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:42.664+0000 7febf67e6700 1 -- 172.21.15.194:0/601971833 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7febf0107ee0 con 0x7febf00f62a0 2024-01-18T06:51:42.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:42.665+0000 7febe5ffb700 1 -- 172.21.15.194:0/601971833 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7febe001c920 con 0x7febf00f62a0 2024-01-18T06:51:42.664 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:51:42.665 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:51:42.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:42.666+0000 7febf67e6700 1 -- 172.21.15.194:0/601971833 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7febdc02ff60 msgr2=0x7febdc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:42.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:42.666+0000 7febf67e6700 1 --2- 172.21.15.194:0/601971833 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7febdc02ff60 0x7febdc032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7febec0096a0 tx=0x7febec006b40).stop 2024-01-18T06:51:42.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:42.666+0000 7febf67e6700 1 -- 172.21.15.194:0/601971833 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7febf00f62a0 msgr2=0x7febf01064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:42.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:42.666+0000 7febf67e6700 1 --2- 172.21.15.194:0/601971833 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7febf00f62a0 0x7febf01064b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7febe0013510 tx=0x7febe000b040).stop 2024-01-18T06:51:42.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:42.666+0000 7febf67e6700 1 -- 172.21.15.194:0/601971833 shutdown_connections 2024-01-18T06:51:42.667 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:42.666+0000 7febf67e6700 1 --2- 172.21.15.194:0/601971833 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7febdc02ff60 0x7febdc032420 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:42.667 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:42.666+0000 7febf67e6700 1 --2- 172.21.15.194:0/601971833 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7febf00f62a0 0x7febf01064b0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:42.667 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:42.666+0000 7febf67e6700 1 -- 172.21.15.194:0/601971833 >> 172.21.15.194:0/601971833 conn(0x7febf00f1aa0 msgr2=0x7febf00f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:42.667 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:42.666+0000 7febf67e6700 1 -- 172.21.15.194:0/601971833 shutdown_connections 2024-01-18T06:51:42.667 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:42.667+0000 7febf67e6700 1 -- 172.21.15.194:0/601971833 wait complete. 2024-01-18T06:51:42.667 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:51:43.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:43 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:51:42.003546+0000 mgr.smithi047.fuhzzr (mgr.14162) 97 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:51:43.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:43 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:51:42.665733+0000 mon.smithi047 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.194:0/601971833' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:51:44.050 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:51:44.050 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:51:45.245 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:45.243+0000 7f1a1a34c700 1 -- 172.21.15.194:0/3204205887 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1a140f51a0 msgr2=0x7f1a140f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:45.245 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:45.243+0000 7f1a1a34c700 1 --2- 172.21.15.194:0/3204205887 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1a140f51a0 0x7f1a140f55c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f1a04009fb0 tx=0x7f1a04009b30).stop 2024-01-18T06:51:45.245 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:45.244+0000 7f1a1a34c700 1 -- 172.21.15.194:0/3204205887 shutdown_connections 2024-01-18T06:51:45.245 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:45.244+0000 7f1a1a34c700 1 --2- 172.21.15.194:0/3204205887 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1a140f51a0 0x7f1a140f55c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:45.245 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:45.244+0000 7f1a1a34c700 1 -- 172.21.15.194:0/3204205887 >> 172.21.15.194:0/3204205887 conn(0x7f1a140f09a0 msgr2=0x7f1a140f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:45.245 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:45.244+0000 7f1a1a34c700 1 -- 172.21.15.194:0/3204205887 shutdown_connections 2024-01-18T06:51:45.245 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:45.244+0000 7f1a1a34c700 1 -- 172.21.15.194:0/3204205887 wait complete. 2024-01-18T06:51:45.246 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:45.244+0000 7f1a1a34c700 1 Processor -- start 2024-01-18T06:51:45.246 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:45.244+0000 7f1a1a34c700 1 -- start start 2024-01-18T06:51:45.246 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:45.244+0000 7f1a1a34c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1a140f51a0 0x7f1a141091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:45.246 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:45.244+0000 7f1a1a34c700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a14060240 con 0x7f1a140f51a0 2024-01-18T06:51:45.246 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:45.245+0000 7f1a1934a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1a140f51a0 0x7f1a141091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:45.246 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:45.245+0000 7f1a1934a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1a140f51a0 0x7f1a141091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:51484/0 (socket says 172.21.15.194:51484) 2024-01-18T06:51:45.246 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:45.245+0000 7f1a1934a700 1 -- 172.21.15.194:0/2694364762 learned_addr learned my addr 172.21.15.194:0/2694364762 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:51:45.246 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:45.245+0000 7f1a1934a700 1 -- 172.21.15.194:0/2694364762 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a04005040 con 0x7f1a140f51a0 2024-01-18T06:51:45.246 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:45.245+0000 7f1a1934a700 1 --2- 172.21.15.194:0/2694364762 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1a140f51a0 0x7f1a141091b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f1a04013510 tx=0x7f1a0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:45.246 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:45.245+0000 7f1a09ffb700 1 -- 172.21.15.194:0/2694364762 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a04009320 con 0x7f1a140f51a0 2024-01-18T06:51:45.246 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:45.245+0000 7f1a1a34c700 1 -- 172.21.15.194:0/2694364762 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a14060440 con 0x7f1a140f51a0 2024-01-18T06:51:45.246 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:45.245+0000 7f1a09ffb700 1 -- 172.21.15.194:0/2694364762 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1a04013f00 con 0x7f1a140f51a0 2024-01-18T06:51:45.246 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:45.245+0000 7f1a09ffb700 1 -- 172.21.15.194:0/2694364762 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a04014dc0 con 0x7f1a140f51a0 2024-01-18T06:51:45.247 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:45.245+0000 7f1a1a34c700 1 -- 172.21.15.194:0/2694364762 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a14109860 con 0x7f1a140f51a0 2024-01-18T06:51:45.247 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:45.246+0000 7f1a1a34c700 1 -- 172.21.15.194:0/2694364762 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a14045ad0 con 0x7f1a140f51a0 2024-01-18T06:51:45.247 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:45.246+0000 7f1a09ffb700 1 -- 172.21.15.194:0/2694364762 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1a040095f0 con 0x7f1a140f51a0 2024-01-18T06:51:45.248 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:45.246+0000 7f1a09ffb700 1 --2- 172.21.15.194:0/2694364762 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f1a0002ff60 0x7f1a00032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:45.248 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:45.246+0000 7f1a09ffb700 1 -- 172.21.15.194:0/2694364762 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1a04041020 con 0x7f1a140f51a0 2024-01-18T06:51:45.248 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:45.247+0000 7f1a18b49700 1 --2- 172.21.15.194:0/2694364762 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f1a0002ff60 0x7f1a00032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:45.248 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:45.247+0000 7f1a18b49700 1 --2- 172.21.15.194:0/2694364762 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f1a0002ff60 0x7f1a00032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f1a100096a0 tx=0x7f1a10006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:45.251 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:45.251+0000 7f1a09ffb700 1 -- 172.21.15.194:0/2694364762 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1a04014420 con 0x7f1a140f51a0 2024-01-18T06:51:45.454 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:45.454+0000 7f1a1a34c700 1 -- 172.21.15.194:0/2694364762 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1a1410a070 con 0x7f1a140f51a0 2024-01-18T06:51:45.454 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:45.455+0000 7f1a09ffb700 1 -- 172.21.15.194:0/2694364762 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1a0401c920 con 0x7f1a140f51a0 2024-01-18T06:51:45.455 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:51:45.455 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:51:45.456 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:45.456+0000 7f1a1a34c700 1 -- 172.21.15.194:0/2694364762 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f1a0002ff60 msgr2=0x7f1a00032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:45.456 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:45.456+0000 7f1a1a34c700 1 --2- 172.21.15.194:0/2694364762 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f1a0002ff60 0x7f1a00032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f1a100096a0 tx=0x7f1a10006b40).stop 2024-01-18T06:51:45.456 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:45.456+0000 7f1a1a34c700 1 -- 172.21.15.194:0/2694364762 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1a140f51a0 msgr2=0x7f1a141091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:45.456 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:45.456+0000 7f1a1a34c700 1 --2- 172.21.15.194:0/2694364762 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1a140f51a0 0x7f1a141091b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f1a04013510 tx=0x7f1a0400b040).stop 2024-01-18T06:51:45.456 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:45.456+0000 7f1a1a34c700 1 -- 172.21.15.194:0/2694364762 shutdown_connections 2024-01-18T06:51:45.456 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:45.456+0000 7f1a1a34c700 1 --2- 172.21.15.194:0/2694364762 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f1a0002ff60 0x7f1a00032420 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:45.456 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:45.456+0000 7f1a1a34c700 1 --2- 172.21.15.194:0/2694364762 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1a140f51a0 0x7f1a141091b0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:45.456 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:45.456+0000 7f1a1a34c700 1 -- 172.21.15.194:0/2694364762 >> 172.21.15.194:0/2694364762 conn(0x7f1a140f09a0 msgr2=0x7f1a140f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:45.457 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:45.456+0000 7f1a1a34c700 1 -- 172.21.15.194:0/2694364762 shutdown_connections 2024-01-18T06:51:45.457 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:45.456+0000 7f1a1a34c700 1 -- 172.21.15.194:0/2694364762 wait complete. 2024-01-18T06:51:45.457 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:51:45.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:45 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:51:44.003887+0000 mgr.smithi047.fuhzzr (mgr.14162) 98 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:51:46.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:46 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:51:45.455842+0000 mon.smithi047 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.194:0/2694364762' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:51:46.847 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:51:46.847 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:51:47.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:47 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:51:46.004190+0000 mgr.smithi047.fuhzzr (mgr.14162) 99 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:51:48.111 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:48.111+0000 7fdef06a5700 1 -- 172.21.15.194:0/3788197931 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdee80f5410 msgr2=0x7fdee80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:48.111 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:48.111+0000 7fdef06a5700 1 --2- 172.21.15.194:0/3788197931 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdee80f5410 0x7fdee80f5830 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7fded8009fb0 tx=0x7fded8009b30).stop 2024-01-18T06:51:48.111 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:48.111+0000 7fdef06a5700 1 -- 172.21.15.194:0/3788197931 shutdown_connections 2024-01-18T06:51:48.111 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:48.111+0000 7fdef06a5700 1 --2- 172.21.15.194:0/3788197931 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdee80f5410 0x7fdee80f5830 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:48.111 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:48.111+0000 7fdef06a5700 1 -- 172.21.15.194:0/3788197931 >> 172.21.15.194:0/3788197931 conn(0x7fdee80f0b70 msgr2=0x7fdee80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:48.112 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:48.111+0000 7fdef06a5700 1 -- 172.21.15.194:0/3788197931 shutdown_connections 2024-01-18T06:51:48.112 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:48.111+0000 7fdef06a5700 1 -- 172.21.15.194:0/3788197931 wait complete. 2024-01-18T06:51:48.112 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:48.112+0000 7fdef06a5700 1 Processor -- start 2024-01-18T06:51:48.112 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:48.112+0000 7fdef06a5700 1 -- start start 2024-01-18T06:51:48.112 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:48.112+0000 7fdef06a5700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdee80f5410 0x7fdee8106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:48.112 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:48.112+0000 7fdef06a5700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee8106c00 con 0x7fdee80f5410 2024-01-18T06:51:48.114 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:48.112+0000 7fdeee441700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdee80f5410 0x7fdee8106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:48.114 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:48.112+0000 7fdeee441700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdee80f5410 0x7fdee8106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:47138/0 (socket says 172.21.15.194:47138) 2024-01-18T06:51:48.114 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:48.112+0000 7fdeee441700 1 -- 172.21.15.194:0/1890117843 learned_addr learned my addr 172.21.15.194:0/1890117843 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:51:48.114 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:48.112+0000 7fdeee441700 1 -- 172.21.15.194:0/1890117843 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded8005040 con 0x7fdee80f5410 2024-01-18T06:51:48.114 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:48.113+0000 7fdeee441700 1 --2- 172.21.15.194:0/1890117843 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdee80f5410 0x7fdee8106710 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fded8014000 tx=0x7fded800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:48.114 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:48.113+0000 7fdedeffd700 1 -- 172.21.15.194:0/1890117843 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fded80095f0 con 0x7fdee80f5410 2024-01-18T06:51:48.114 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:48.113+0000 7fdef06a5700 1 -- 172.21.15.194:0/1890117843 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdee8106e00 con 0x7fdee80f5410 2024-01-18T06:51:48.114 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:48.113+0000 7fdedeffd700 1 -- 172.21.15.194:0/1890117843 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fded8007e70 con 0x7fdee80f5410 2024-01-18T06:51:48.114 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:48.113+0000 7fdedeffd700 1 -- 172.21.15.194:0/1890117843 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fded8014ea0 con 0x7fdee80f5410 2024-01-18T06:51:48.114 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:48.113+0000 7fdef06a5700 1 -- 172.21.15.194:0/1890117843 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdee8107290 con 0x7fdee80f5410 2024-01-18T06:51:48.114 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:48.114+0000 7fdedeffd700 1 -- 172.21.15.194:0/1890117843 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fded8025070 con 0x7fdee80f5410 2024-01-18T06:51:48.114 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:48.114+0000 7fdef06a5700 1 -- 172.21.15.194:0/1890117843 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdee8045ad0 con 0x7fdee80f5410 2024-01-18T06:51:48.115 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:48.114+0000 7fdedeffd700 1 --2- 172.21.15.194:0/1890117843 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fded402ff60 0x7fded4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:48.115 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:48.114+0000 7fdedeffd700 1 -- 172.21.15.194:0/1890117843 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fded80409f0 con 0x7fdee80f5410 2024-01-18T06:51:48.115 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:48.114+0000 7fdeedc40700 1 --2- 172.21.15.194:0/1890117843 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fded402ff60 0x7fded4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:48.115 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:48.114+0000 7fdeedc40700 1 --2- 172.21.15.194:0/1890117843 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fded402ff60 0x7fded4032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fdee40096a0 tx=0x7fdee4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:48.118 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:48.119+0000 7fdedeffd700 1 -- 172.21.15.194:0/1890117843 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fded80079c0 con 0x7fdee80f5410 2024-01-18T06:51:48.323 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:48.324+0000 7fdef06a5700 1 -- 172.21.15.194:0/1890117843 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdee8108140 con 0x7fdee80f5410 2024-01-18T06:51:48.324 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:48.324+0000 7fdedeffd700 1 -- 172.21.15.194:0/1890117843 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fded800f300 con 0x7fdee80f5410 2024-01-18T06:51:48.324 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:51:48.324 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:51:48.326 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:48.326+0000 7fdef06a5700 1 -- 172.21.15.194:0/1890117843 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fded402ff60 msgr2=0x7fded4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:48.326 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:48.326+0000 7fdef06a5700 1 --2- 172.21.15.194:0/1890117843 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fded402ff60 0x7fded4032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fdee40096a0 tx=0x7fdee4006b40).stop 2024-01-18T06:51:48.326 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:48.326+0000 7fdef06a5700 1 -- 172.21.15.194:0/1890117843 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdee80f5410 msgr2=0x7fdee8106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:48.326 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:48.326+0000 7fdef06a5700 1 --2- 172.21.15.194:0/1890117843 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdee80f5410 0x7fdee8106710 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fded8014000 tx=0x7fded800b040).stop 2024-01-18T06:51:48.326 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:48.326+0000 7fdef06a5700 1 -- 172.21.15.194:0/1890117843 shutdown_connections 2024-01-18T06:51:48.326 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:48.326+0000 7fdef06a5700 1 --2- 172.21.15.194:0/1890117843 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fded402ff60 0x7fded4032420 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:48.326 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:48.326+0000 7fdef06a5700 1 --2- 172.21.15.194:0/1890117843 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdee80f5410 0x7fdee8106710 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:48.326 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:48.326+0000 7fdef06a5700 1 -- 172.21.15.194:0/1890117843 >> 172.21.15.194:0/1890117843 conn(0x7fdee80f0b70 msgr2=0x7fdee805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:48.326 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:48.326+0000 7fdef06a5700 1 -- 172.21.15.194:0/1890117843 shutdown_connections 2024-01-18T06:51:48.326 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:48.326+0000 7fdef06a5700 1 -- 172.21.15.194:0/1890117843 wait complete. 2024-01-18T06:51:48.327 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:51:49.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:48 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:51:48.004564+0000 mgr.smithi047.fuhzzr (mgr.14162) 100 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:51:49.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:48 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:51:48.325397+0000 mon.smithi047 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.194:0/1890117843' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:51:49.713 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:51:49.713 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:51:50.800 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:50.799+0000 7f47aa0aa700 1 -- 172.21.15.194:0/16622496 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f47a40f5410 msgr2=0x7f47a40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:50.800 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:50.799+0000 7f47aa0aa700 1 --2- 172.21.15.194:0/16622496 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f47a40f5410 0x7f47a40f5830 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f478c009fb0 tx=0x7f478c009b30).stop 2024-01-18T06:51:50.800 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:50.799+0000 7f47aa0aa700 1 -- 172.21.15.194:0/16622496 shutdown_connections 2024-01-18T06:51:50.800 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:50.799+0000 7f47aa0aa700 1 --2- 172.21.15.194:0/16622496 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f47a40f5410 0x7f47a40f5830 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:50.800 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:50.799+0000 7f47aa0aa700 1 -- 172.21.15.194:0/16622496 >> 172.21.15.194:0/16622496 conn(0x7f47a40f0b70 msgr2=0x7f47a40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:50.800 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:50.799+0000 7f47aa0aa700 1 -- 172.21.15.194:0/16622496 shutdown_connections 2024-01-18T06:51:50.800 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:50.799+0000 7f47aa0aa700 1 -- 172.21.15.194:0/16622496 wait complete. 2024-01-18T06:51:50.800 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:50.800+0000 7f47aa0aa700 1 Processor -- start 2024-01-18T06:51:50.800 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:50.800+0000 7f47aa0aa700 1 -- start start 2024-01-18T06:51:50.801 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:50.800+0000 7f47aa0aa700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f47a40f5410 0x7f47a4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:50.801 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:50.800+0000 7f47aa0aa700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a4105b20 con 0x7f47a40f5410 2024-01-18T06:51:50.801 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:50.800+0000 7f47a37fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f47a40f5410 0x7f47a4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:50.801 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:50.800+0000 7f47a37fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f47a40f5410 0x7f47a4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:47150/0 (socket says 172.21.15.194:47150) 2024-01-18T06:51:50.801 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:50.800+0000 7f47a37fe700 1 -- 172.21.15.194:0/1433958670 learned_addr learned my addr 172.21.15.194:0/1433958670 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:51:50.802 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:50.801+0000 7f47a37fe700 1 -- 172.21.15.194:0/1433958670 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f478c005040 con 0x7f47a40f5410 2024-01-18T06:51:50.802 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:50.801+0000 7f47a37fe700 1 --2- 172.21.15.194:0/1433958670 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f47a40f5410 0x7f47a4105630 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f478c013510 tx=0x7f478c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:50.802 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:50.801+0000 7f479bfff700 1 -- 172.21.15.194:0/1433958670 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f478c009320 con 0x7f47a40f5410 2024-01-18T06:51:50.802 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:50.801+0000 7f47aa0aa700 1 -- 172.21.15.194:0/1433958670 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47a4105d20 con 0x7f47a40f5410 2024-01-18T06:51:50.802 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:50.801+0000 7f479bfff700 1 -- 172.21.15.194:0/1433958670 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f478c013f00 con 0x7f47a40f5410 2024-01-18T06:51:50.802 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:50.801+0000 7f479bfff700 1 -- 172.21.15.194:0/1433958670 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f478c014d90 con 0x7f47a40f5410 2024-01-18T06:51:50.803 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:50.801+0000 7f47aa0aa700 1 -- 172.21.15.194:0/1433958670 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47a41061b0 con 0x7f47a40f5410 2024-01-18T06:51:50.803 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:50.802+0000 7f479bfff700 1 -- 172.21.15.194:0/1433958670 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f478c0095f0 con 0x7f47a40f5410 2024-01-18T06:51:50.803 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:50.802+0000 7f47aa0aa700 1 -- 172.21.15.194:0/1433958670 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47a4045ad0 con 0x7f47a40f5410 2024-01-18T06:51:50.803 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:50.802+0000 7f479bfff700 1 --2- 172.21.15.194:0/1433958670 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f479002ff60 0x7f4790032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:50.803 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:50.802+0000 7f479bfff700 1 -- 172.21.15.194:0/1433958670 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f478c040fe0 con 0x7f47a40f5410 2024-01-18T06:51:50.803 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:50.802+0000 7f47a2ffd700 1 --2- 172.21.15.194:0/1433958670 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f479002ff60 0x7f4790032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:50.803 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:50.803+0000 7f47a2ffd700 1 --2- 172.21.15.194:0/1433958670 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f479002ff60 0x7f4790032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f47940096a0 tx=0x7f4794006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:50.806 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:50.807+0000 7f479bfff700 1 -- 172.21.15.194:0/1433958670 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f478c01d7f0 con 0x7f47a40f5410 2024-01-18T06:51:51.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:51.015+0000 7f47aa0aa700 1 -- 172.21.15.194:0/1433958670 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f47a40fae60 con 0x7f47a40f5410 2024-01-18T06:51:51.015 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:51.016+0000 7f479bfff700 1 -- 172.21.15.194:0/1433958670 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f478c010230 con 0x7f47a40f5410 2024-01-18T06:51:51.015 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:51:51.015 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:51:51.017 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:51.017+0000 7f47aa0aa700 1 -- 172.21.15.194:0/1433958670 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f479002ff60 msgr2=0x7f4790032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:51.017 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:51.017+0000 7f47aa0aa700 1 --2- 172.21.15.194:0/1433958670 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f479002ff60 0x7f4790032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f47940096a0 tx=0x7f4794006b40).stop 2024-01-18T06:51:51.017 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:51.017+0000 7f47aa0aa700 1 -- 172.21.15.194:0/1433958670 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f47a40f5410 msgr2=0x7f47a4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:51.017 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:51.017+0000 7f47aa0aa700 1 --2- 172.21.15.194:0/1433958670 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f47a40f5410 0x7f47a4105630 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f478c013510 tx=0x7f478c00b040).stop 2024-01-18T06:51:51.017 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:51.017+0000 7f47aa0aa700 1 -- 172.21.15.194:0/1433958670 shutdown_connections 2024-01-18T06:51:51.017 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:51.017+0000 7f47aa0aa700 1 --2- 172.21.15.194:0/1433958670 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f479002ff60 0x7f4790032420 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:51.017 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:51.017+0000 7f47aa0aa700 1 --2- 172.21.15.194:0/1433958670 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f47a40f5410 0x7f47a4105630 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:51.017 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:51.017+0000 7f47aa0aa700 1 -- 172.21.15.194:0/1433958670 >> 172.21.15.194:0/1433958670 conn(0x7f47a40f0b70 msgr2=0x7f47a40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:51.017 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:51.017+0000 7f47aa0aa700 1 -- 172.21.15.194:0/1433958670 shutdown_connections 2024-01-18T06:51:51.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:51.017+0000 7f47aa0aa700 1 -- 172.21.15.194:0/1433958670 wait complete. 2024-01-18T06:51:51.018 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:51:51.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:51 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:51:50.004927+0000 mgr.smithi047.fuhzzr (mgr.14162) 101 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:51:51.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:51 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:51:51.016599+0000 mon.smithi047 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.194:0/1433958670' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:51:52.403 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:51:52.403 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:51:53.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:53 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:51:52.005238+0000 mgr.smithi047.fuhzzr (mgr.14162) 102 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:51:53.589 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:53.587+0000 7fa1848e1700 1 -- 172.21.15.194:0/3116617329 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1800f5410 msgr2=0x7fa1800f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:53.589 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:53.587+0000 7fa1848e1700 1 --2- 172.21.15.194:0/3116617329 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1800f5410 0x7fa1800f5830 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fa168009fb0 tx=0x7fa168009b30).stop 2024-01-18T06:51:53.589 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:53.587+0000 7fa1848e1700 1 -- 172.21.15.194:0/3116617329 shutdown_connections 2024-01-18T06:51:53.589 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:53.587+0000 7fa1848e1700 1 --2- 172.21.15.194:0/3116617329 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1800f5410 0x7fa1800f5830 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:53.589 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:53.587+0000 7fa1848e1700 1 -- 172.21.15.194:0/3116617329 >> 172.21.15.194:0/3116617329 conn(0x7fa1800f0b70 msgr2=0x7fa1800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:53.589 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:53.587+0000 7fa1848e1700 1 -- 172.21.15.194:0/3116617329 shutdown_connections 2024-01-18T06:51:53.589 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:53.587+0000 7fa1848e1700 1 -- 172.21.15.194:0/3116617329 wait complete. 2024-01-18T06:51:53.589 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:53.588+0000 7fa1848e1700 1 Processor -- start 2024-01-18T06:51:53.589 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:53.588+0000 7fa1848e1700 1 -- start start 2024-01-18T06:51:53.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:53.588+0000 7fa1848e1700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1800f5410 0x7fa18010b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:53.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:53.588+0000 7fa1848e1700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa180107240 con 0x7fa1800f5410 2024-01-18T06:51:53.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:53.588+0000 7fa17e59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1800f5410 0x7fa18010b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:53.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:53.588+0000 7fa17e59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1800f5410 0x7fa18010b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:47156/0 (socket says 172.21.15.194:47156) 2024-01-18T06:51:53.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:53.588+0000 7fa17e59c700 1 -- 172.21.15.194:0/1416857483 learned_addr learned my addr 172.21.15.194:0/1416857483 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:51:53.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:53.588+0000 7fa17e59c700 1 -- 172.21.15.194:0/1416857483 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa168005040 con 0x7fa1800f5410 2024-01-18T06:51:53.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:53.589+0000 7fa17e59c700 1 --2- 172.21.15.194:0/1416857483 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1800f5410 0x7fa18010b850 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fa168014000 tx=0x7fa16800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:53.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:53.589+0000 7fa176ffd700 1 -- 172.21.15.194:0/1416857483 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1680095f0 con 0x7fa1800f5410 2024-01-18T06:51:53.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:53.589+0000 7fa1848e1700 1 -- 172.21.15.194:0/1416857483 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa180060220 con 0x7fa1800f5410 2024-01-18T06:51:53.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:53.589+0000 7fa176ffd700 1 -- 172.21.15.194:0/1416857483 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa168007e70 con 0x7fa1800f5410 2024-01-18T06:51:53.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:53.589+0000 7fa176ffd700 1 -- 172.21.15.194:0/1416857483 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa168014ea0 con 0x7fa1800f5410 2024-01-18T06:51:53.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:53.589+0000 7fa1848e1700 1 -- 172.21.15.194:0/1416857483 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa18010bd40 con 0x7fa1800f5410 2024-01-18T06:51:53.592 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:53.590+0000 7fa176ffd700 1 -- 172.21.15.194:0/1416857483 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa168025070 con 0x7fa1800f5410 2024-01-18T06:51:53.592 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:53.590+0000 7fa1848e1700 1 -- 172.21.15.194:0/1416857483 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa180045ad0 con 0x7fa1800f5410 2024-01-18T06:51:53.592 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:53.590+0000 7fa176ffd700 1 --2- 172.21.15.194:0/1416857483 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa16c02ff60 0x7fa16c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:53.592 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:53.590+0000 7fa176ffd700 1 -- 172.21.15.194:0/1416857483 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa168040970 con 0x7fa1800f5410 2024-01-18T06:51:53.592 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:53.590+0000 7fa17dd9b700 1 --2- 172.21.15.194:0/1416857483 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa16c02ff60 0x7fa16c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:53.592 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:53.590+0000 7fa17dd9b700 1 --2- 172.21.15.194:0/1416857483 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa16c02ff60 0x7fa16c032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fa1700096a0 tx=0x7fa170006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:53.594 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:53.594+0000 7fa176ffd700 1 -- 172.21.15.194:0/1416857483 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa1680079c0 con 0x7fa1800f5410 2024-01-18T06:51:53.799 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:53.799+0000 7fa1848e1700 1 -- 172.21.15.194:0/1416857483 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa1800fae60 con 0x7fa1800f5410 2024-01-18T06:51:53.799 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:53.800+0000 7fa176ffd700 1 -- 172.21.15.194:0/1416857483 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa16800f300 con 0x7fa1800f5410 2024-01-18T06:51:53.799 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:51:53.800 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:51:53.801 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:53.801+0000 7fa1848e1700 1 -- 172.21.15.194:0/1416857483 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa16c02ff60 msgr2=0x7fa16c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:53.801 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:53.801+0000 7fa1848e1700 1 --2- 172.21.15.194:0/1416857483 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa16c02ff60 0x7fa16c032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fa1700096a0 tx=0x7fa170006b40).stop 2024-01-18T06:51:53.801 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:53.801+0000 7fa1848e1700 1 -- 172.21.15.194:0/1416857483 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1800f5410 msgr2=0x7fa18010b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:53.802 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:53.801+0000 7fa1848e1700 1 --2- 172.21.15.194:0/1416857483 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1800f5410 0x7fa18010b850 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fa168014000 tx=0x7fa16800b040).stop 2024-01-18T06:51:53.802 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:53.801+0000 7fa1848e1700 1 -- 172.21.15.194:0/1416857483 shutdown_connections 2024-01-18T06:51:53.802 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:53.801+0000 7fa1848e1700 1 --2- 172.21.15.194:0/1416857483 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa16c02ff60 0x7fa16c032420 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:53.802 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:53.801+0000 7fa1848e1700 1 --2- 172.21.15.194:0/1416857483 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa1800f5410 0x7fa18010b850 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:53.802 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:53.801+0000 7fa1848e1700 1 -- 172.21.15.194:0/1416857483 >> 172.21.15.194:0/1416857483 conn(0x7fa1800f0b70 msgr2=0x7fa1800f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:53.802 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:53.802+0000 7fa1848e1700 1 -- 172.21.15.194:0/1416857483 shutdown_connections 2024-01-18T06:51:53.802 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:53.802+0000 7fa1848e1700 1 -- 172.21.15.194:0/1416857483 wait complete. 2024-01-18T06:51:53.802 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:51:54.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:54 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:51:53.800839+0000 mon.smithi047 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.194:0/1416857483' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:51:55.185 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:51:55.186 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:51:55.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:55 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:51:54.005558+0000 mgr.smithi047.fuhzzr (mgr.14162) 103 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:51:56.304 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:56.303+0000 7fc6cb42e700 1 -- 172.21.15.194:0/1669995758 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc6c40f5410 msgr2=0x7fc6c40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:56.305 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:56.303+0000 7fc6cb42e700 1 --2- 172.21.15.194:0/1669995758 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc6c40f5410 0x7fc6c40f5830 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7fc6b8009fb0 tx=0x7fc6b8009b30).stop 2024-01-18T06:51:56.305 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:56.303+0000 7fc6cb42e700 1 -- 172.21.15.194:0/1669995758 shutdown_connections 2024-01-18T06:51:56.305 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:56.303+0000 7fc6cb42e700 1 --2- 172.21.15.194:0/1669995758 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc6c40f5410 0x7fc6c40f5830 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:56.305 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:56.303+0000 7fc6cb42e700 1 -- 172.21.15.194:0/1669995758 >> 172.21.15.194:0/1669995758 conn(0x7fc6c40f0b70 msgr2=0x7fc6c40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:56.305 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:56.303+0000 7fc6cb42e700 1 -- 172.21.15.194:0/1669995758 shutdown_connections 2024-01-18T06:51:56.305 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:56.303+0000 7fc6cb42e700 1 -- 172.21.15.194:0/1669995758 wait complete. 2024-01-18T06:51:56.305 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:56.304+0000 7fc6cb42e700 1 Processor -- start 2024-01-18T06:51:56.305 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:56.304+0000 7fc6cb42e700 1 -- start start 2024-01-18T06:51:56.305 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:56.304+0000 7fc6cb42e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc6c40f5410 0x7fc6c4109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:56.305 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:56.304+0000 7fc6cb42e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c4060240 con 0x7fc6c40f5410 2024-01-18T06:51:56.305 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:56.304+0000 7fc6c91ca700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc6c40f5410 0x7fc6c4109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:56.305 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:56.304+0000 7fc6c91ca700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc6c40f5410 0x7fc6c4109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:47164/0 (socket says 172.21.15.194:47164) 2024-01-18T06:51:56.305 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:56.304+0000 7fc6c91ca700 1 -- 172.21.15.194:0/4219933251 learned_addr learned my addr 172.21.15.194:0/4219933251 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:51:56.306 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:56.305+0000 7fc6c91ca700 1 -- 172.21.15.194:0/4219933251 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b8005040 con 0x7fc6c40f5410 2024-01-18T06:51:56.306 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:56.305+0000 7fc6c91ca700 1 --2- 172.21.15.194:0/4219933251 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc6c40f5410 0x7fc6c4109430 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fc6b8013510 tx=0x7fc6b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:56.306 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:56.305+0000 7fc6b5ffb700 1 -- 172.21.15.194:0/4219933251 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc6b8009320 con 0x7fc6c40f5410 2024-01-18T06:51:56.306 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:56.305+0000 7fc6cb42e700 1 -- 172.21.15.194:0/4219933251 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6c4060440 con 0x7fc6c40f5410 2024-01-18T06:51:56.306 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:56.305+0000 7fc6b5ffb700 1 -- 172.21.15.194:0/4219933251 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc6b8013f00 con 0x7fc6c40f5410 2024-01-18T06:51:56.306 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:56.305+0000 7fc6b5ffb700 1 -- 172.21.15.194:0/4219933251 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc6b8014d90 con 0x7fc6c40f5410 2024-01-18T06:51:56.306 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:56.305+0000 7fc6cb42e700 1 -- 172.21.15.194:0/4219933251 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6c4109ae0 con 0x7fc6c40f5410 2024-01-18T06:51:56.307 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:56.306+0000 7fc6b5ffb700 1 -- 172.21.15.194:0/4219933251 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc6b80263a0 con 0x7fc6c40f5410 2024-01-18T06:51:56.307 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:56.306+0000 7fc6cb42e700 1 -- 172.21.15.194:0/4219933251 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6c4045ad0 con 0x7fc6c40f5410 2024-01-18T06:51:56.307 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:56.306+0000 7fc6b5ffb700 1 --2- 172.21.15.194:0/4219933251 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc6b002ff60 0x7fc6b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:56.307 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:56.306+0000 7fc6b5ffb700 1 -- 172.21.15.194:0/4219933251 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc6b8041000 con 0x7fc6c40f5410 2024-01-18T06:51:56.307 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:56.306+0000 7fc6c89c9700 1 --2- 172.21.15.194:0/4219933251 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc6b002ff60 0x7fc6b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:56.307 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:56.307+0000 7fc6c89c9700 1 --2- 172.21.15.194:0/4219933251 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc6b002ff60 0x7fc6b0032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fc6c00096a0 tx=0x7fc6c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:56.310 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:56.311+0000 7fc6b5ffb700 1 -- 172.21.15.194:0/4219933251 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc6b80126f0 con 0x7fc6c40f5410 2024-01-18T06:51:56.519 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:56.519+0000 7fc6cb42e700 1 -- 172.21.15.194:0/4219933251 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc6c40fae60 con 0x7fc6c40f5410 2024-01-18T06:51:56.519 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:56.520+0000 7fc6b5ffb700 1 -- 172.21.15.194:0/4219933251 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc6b800f340 con 0x7fc6c40f5410 2024-01-18T06:51:56.520 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:51:56.520 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:51:56.521 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:56.522+0000 7fc6cb42e700 1 -- 172.21.15.194:0/4219933251 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc6b002ff60 msgr2=0x7fc6b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:56.522 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:56.522+0000 7fc6cb42e700 1 --2- 172.21.15.194:0/4219933251 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc6b002ff60 0x7fc6b0032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fc6c00096a0 tx=0x7fc6c0006b40).stop 2024-01-18T06:51:56.522 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:56.522+0000 7fc6cb42e700 1 -- 172.21.15.194:0/4219933251 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc6c40f5410 msgr2=0x7fc6c4109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:56.522 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:56.522+0000 7fc6cb42e700 1 --2- 172.21.15.194:0/4219933251 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc6c40f5410 0x7fc6c4109430 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fc6b8013510 tx=0x7fc6b800b040).stop 2024-01-18T06:51:56.522 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:56.522+0000 7fc6cb42e700 1 -- 172.21.15.194:0/4219933251 shutdown_connections 2024-01-18T06:51:56.522 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:56.522+0000 7fc6cb42e700 1 --2- 172.21.15.194:0/4219933251 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc6b002ff60 0x7fc6b0032420 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:56.522 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:56.522+0000 7fc6cb42e700 1 --2- 172.21.15.194:0/4219933251 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc6c40f5410 0x7fc6c4109430 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:56.522 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:56.522+0000 7fc6cb42e700 1 -- 172.21.15.194:0/4219933251 >> 172.21.15.194:0/4219933251 conn(0x7fc6c40f0b70 msgr2=0x7fc6c40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:56.522 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:56.522+0000 7fc6cb42e700 1 -- 172.21.15.194:0/4219933251 shutdown_connections 2024-01-18T06:51:56.522 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:56.522+0000 7fc6cb42e700 1 -- 172.21.15.194:0/4219933251 wait complete. 2024-01-18T06:51:56.522 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:51:57.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:57 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:51:56.005881+0000 mgr.smithi047.fuhzzr (mgr.14162) 104 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:51:57.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:57 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:51:56.521038+0000 mon.smithi047 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.194:0/4219933251' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:51:57.863 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:51:57.864 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:51:59.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:59.048+0000 7fe1d6b54700 1 -- 172.21.15.194:0/143310145 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1d0060280 msgr2=0x7fe1d00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:59.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:59.048+0000 7fe1d6b54700 1 --2- 172.21.15.194:0/143310145 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1d0060280 0x7fe1d00f90f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fe1c4009fb0 tx=0x7fe1c4009b30).stop 2024-01-18T06:51:59.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:59.048+0000 7fe1d6b54700 1 -- 172.21.15.194:0/143310145 shutdown_connections 2024-01-18T06:51:59.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:59.048+0000 7fe1d6b54700 1 --2- 172.21.15.194:0/143310145 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1d0060280 0x7fe1d00f90f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:59.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:59.048+0000 7fe1d6b54700 1 -- 172.21.15.194:0/143310145 >> 172.21.15.194:0/143310145 conn(0x7fe1d00f0b70 msgr2=0x7fe1d00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:59.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:59.048+0000 7fe1d6b54700 1 -- 172.21.15.194:0/143310145 shutdown_connections 2024-01-18T06:51:59.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:59.048+0000 7fe1d6b54700 1 -- 172.21.15.194:0/143310145 wait complete. 2024-01-18T06:51:59.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:59.049+0000 7fe1d6b54700 1 Processor -- start 2024-01-18T06:51:59.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:59.049+0000 7fe1d6b54700 1 -- start start 2024-01-18T06:51:59.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:59.049+0000 7fe1d6b54700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1d0060280 0x7fe1d01066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:59.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:59.049+0000 7fe1d6b54700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d0106bd0 con 0x7fe1d0060280 2024-01-18T06:51:59.050 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:59.049+0000 7fe1d48f0700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1d0060280 0x7fe1d01066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:59.050 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:59.049+0000 7fe1d48f0700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1d0060280 0x7fe1d01066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:53822/0 (socket says 172.21.15.194:53822) 2024-01-18T06:51:59.050 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:59.049+0000 7fe1d48f0700 1 -- 172.21.15.194:0/1616861912 learned_addr learned my addr 172.21.15.194:0/1616861912 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:51:59.051 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:59.050+0000 7fe1d48f0700 1 -- 172.21.15.194:0/1616861912 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c4005040 con 0x7fe1d0060280 2024-01-18T06:51:59.051 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:59.050+0000 7fe1d48f0700 1 --2- 172.21.15.194:0/1616861912 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1d0060280 0x7fe1d01066e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fe1c4013510 tx=0x7fe1c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:59.051 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:59.050+0000 7fe1cd7fa700 1 -- 172.21.15.194:0/1616861912 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe1c4009320 con 0x7fe1d0060280 2024-01-18T06:51:59.051 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:59.050+0000 7fe1d6b54700 1 -- 172.21.15.194:0/1616861912 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1d0106dd0 con 0x7fe1d0060280 2024-01-18T06:51:59.051 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:59.050+0000 7fe1cd7fa700 1 -- 172.21.15.194:0/1616861912 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe1c4013f00 con 0x7fe1d0060280 2024-01-18T06:51:59.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:59.050+0000 7fe1cd7fa700 1 -- 172.21.15.194:0/1616861912 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe1c4014dc0 con 0x7fe1d0060280 2024-01-18T06:51:59.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:59.050+0000 7fe1d6b54700 1 -- 172.21.15.194:0/1616861912 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1d0107260 con 0x7fe1d0060280 2024-01-18T06:51:59.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:59.051+0000 7fe1cd7fa700 1 -- 172.21.15.194:0/1616861912 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe1c401d3d0 con 0x7fe1d0060280 2024-01-18T06:51:59.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:59.051+0000 7fe1cd7fa700 1 --2- 172.21.15.194:0/1616861912 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fe1c002ff60 0x7fe1c0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:51:59.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:59.051+0000 7fe1cd7fa700 1 -- 172.21.15.194:0/1616861912 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe1c4041050 con 0x7fe1d0060280 2024-01-18T06:51:59.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:59.051+0000 7fe1d6b54700 1 -- 172.21.15.194:0/1616861912 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1d0045ad0 con 0x7fe1d0060280 2024-01-18T06:51:59.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:59.051+0000 7fe1cffff700 1 --2- 172.21.15.194:0/1616861912 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fe1c002ff60 0x7fe1c0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:51:59.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:59.051+0000 7fe1cffff700 1 --2- 172.21.15.194:0/1616861912 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fe1c002ff60 0x7fe1c0032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fe1bc0096a0 tx=0x7fe1bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:51:59.055 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:59.056+0000 7fe1cd7fa700 1 -- 172.21.15.194:0/1616861912 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe1c4012920 con 0x7fe1d0060280 2024-01-18T06:51:59.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:58 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:51:58.006193+0000 mgr.smithi047.fuhzzr (mgr.14162) 105 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:51:59.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:59.264+0000 7fe1d6b54700 1 -- 172.21.15.194:0/1616861912 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe1d0107f10 con 0x7fe1d0060280 2024-01-18T06:51:59.264 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:59.264+0000 7fe1cd7fa700 1 -- 172.21.15.194:0/1616861912 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe1c402b020 con 0x7fe1d0060280 2024-01-18T06:51:59.264 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:51:59.264 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:51:59.265 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:59.266+0000 7fe1d6b54700 1 -- 172.21.15.194:0/1616861912 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fe1c002ff60 msgr2=0x7fe1c0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:59.265 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:59.266+0000 7fe1d6b54700 1 --2- 172.21.15.194:0/1616861912 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fe1c002ff60 0x7fe1c0032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fe1bc0096a0 tx=0x7fe1bc006b40).stop 2024-01-18T06:51:59.266 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:59.266+0000 7fe1d6b54700 1 -- 172.21.15.194:0/1616861912 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1d0060280 msgr2=0x7fe1d01066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:51:59.266 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:59.266+0000 7fe1d6b54700 1 --2- 172.21.15.194:0/1616861912 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1d0060280 0x7fe1d01066e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fe1c4013510 tx=0x7fe1c400b040).stop 2024-01-18T06:51:59.266 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:59.266+0000 7fe1d6b54700 1 -- 172.21.15.194:0/1616861912 shutdown_connections 2024-01-18T06:51:59.266 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:59.266+0000 7fe1d6b54700 1 --2- 172.21.15.194:0/1616861912 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fe1c002ff60 0x7fe1c0032420 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:59.266 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:59.266+0000 7fe1d6b54700 1 --2- 172.21.15.194:0/1616861912 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe1d0060280 0x7fe1d01066e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:51:59.266 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:59.266+0000 7fe1d6b54700 1 -- 172.21.15.194:0/1616861912 >> 172.21.15.194:0/1616861912 conn(0x7fe1d00f0b70 msgr2=0x7fe1d00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:51:59.266 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:59.266+0000 7fe1d6b54700 1 -- 172.21.15.194:0/1616861912 shutdown_connections 2024-01-18T06:51:59.266 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:51:59.266+0000 7fe1d6b54700 1 -- 172.21.15.194:0/1616861912 wait complete. 2024-01-18T06:51:59.266 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:52:00.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:51:59 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:51:59.265453+0000 mon.smithi047 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.194:0/1616861912' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:52:00.650 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:52:00.650 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:52:01.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:00 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:52:00.006566+0000 mgr.smithi047.fuhzzr (mgr.14162) 106 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:52:01.871 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:01.870+0000 7fc0ec42a700 1 -- 172.21.15.194:0/1504509529 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc0e40f6510 msgr2=0x7fc0e40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:01.871 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:01.870+0000 7fc0ec42a700 1 --2- 172.21.15.194:0/1504509529 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc0e40f6510 0x7fc0e40f6930 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7fc0d4009fb0 tx=0x7fc0d4009b30).stop 2024-01-18T06:52:01.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:01.870+0000 7fc0ec42a700 1 -- 172.21.15.194:0/1504509529 shutdown_connections 2024-01-18T06:52:01.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:01.870+0000 7fc0ec42a700 1 --2- 172.21.15.194:0/1504509529 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc0e40f6510 0x7fc0e40f6930 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:01.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:01.870+0000 7fc0ec42a700 1 -- 172.21.15.194:0/1504509529 >> 172.21.15.194:0/1504509529 conn(0x7fc0e40f1c70 msgr2=0x7fc0e40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:52:01.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:01.871+0000 7fc0ec42a700 1 -- 172.21.15.194:0/1504509529 shutdown_connections 2024-01-18T06:52:01.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:01.871+0000 7fc0ec42a700 1 -- 172.21.15.194:0/1504509529 wait complete. 2024-01-18T06:52:01.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:01.871+0000 7fc0ec42a700 1 Processor -- start 2024-01-18T06:52:01.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:01.871+0000 7fc0ec42a700 1 -- start start 2024-01-18T06:52:01.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:01.871+0000 7fc0ec42a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc0e40f6510 0x7fc0e4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:52:01.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:01.871+0000 7fc0ec42a700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e4106c20 con 0x7fc0e40f6510 2024-01-18T06:52:01.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:01.871+0000 7fc0ea1c6700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc0e40f6510 0x7fc0e4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:52:01.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:01.871+0000 7fc0ea1c6700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc0e40f6510 0x7fc0e4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:53838/0 (socket says 172.21.15.194:53838) 2024-01-18T06:52:01.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:01.871+0000 7fc0ea1c6700 1 -- 172.21.15.194:0/1395729872 learned_addr learned my addr 172.21.15.194:0/1395729872 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:52:01.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:01.872+0000 7fc0ea1c6700 1 -- 172.21.15.194:0/1395729872 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d4005040 con 0x7fc0e40f6510 2024-01-18T06:52:01.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:01.872+0000 7fc0ea1c6700 1 --2- 172.21.15.194:0/1395729872 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc0e40f6510 0x7fc0e4106730 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fc0d4013510 tx=0x7fc0d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:52:01.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:01.872+0000 7fc0daffd700 1 -- 172.21.15.194:0/1395729872 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0d4009320 con 0x7fc0e40f6510 2024-01-18T06:52:01.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:01.872+0000 7fc0ec42a700 1 -- 172.21.15.194:0/1395729872 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0e4106e20 con 0x7fc0e40f6510 2024-01-18T06:52:01.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:01.872+0000 7fc0daffd700 1 -- 172.21.15.194:0/1395729872 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc0d4013f00 con 0x7fc0e40f6510 2024-01-18T06:52:01.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:01.872+0000 7fc0daffd700 1 -- 172.21.15.194:0/1395729872 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0d4014d90 con 0x7fc0e40f6510 2024-01-18T06:52:01.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:01.872+0000 7fc0ec42a700 1 -- 172.21.15.194:0/1395729872 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0e41072b0 con 0x7fc0e40f6510 2024-01-18T06:52:01.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:01.873+0000 7fc0ec42a700 1 -- 172.21.15.194:0/1395729872 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0e4045ad0 con 0x7fc0e40f6510 2024-01-18T06:52:01.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:01.873+0000 7fc0daffd700 1 -- 172.21.15.194:0/1395729872 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc0d4014420 con 0x7fc0e40f6510 2024-01-18T06:52:01.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:01.873+0000 7fc0daffd700 1 --2- 172.21.15.194:0/1395729872 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc0d002ff60 0x7fc0d0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:52:01.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:01.873+0000 7fc0daffd700 1 -- 172.21.15.194:0/1395729872 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc0d4041080 con 0x7fc0e40f6510 2024-01-18T06:52:01.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:01.873+0000 7fc0e99c5700 1 --2- 172.21.15.194:0/1395729872 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc0d002ff60 0x7fc0d0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:52:01.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:01.874+0000 7fc0e99c5700 1 --2- 172.21.15.194:0/1395729872 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc0d002ff60 0x7fc0d0032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fc0e00096a0 tx=0x7fc0e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:52:01.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:01.878+0000 7fc0daffd700 1 -- 172.21.15.194:0/1395729872 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc0d40095f0 con 0x7fc0e40f6510 2024-01-18T06:52:02.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:02.083+0000 7fc0ec42a700 1 -- 172.21.15.194:0/1395729872 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc0e40fbf60 con 0x7fc0e40f6510 2024-01-18T06:52:02.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:02.084+0000 7fc0daffd700 1 -- 172.21.15.194:0/1395729872 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc0d401db70 con 0x7fc0e40f6510 2024-01-18T06:52:02.083 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:52:02.083 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:52:02.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:02.085+0000 7fc0ec42a700 1 -- 172.21.15.194:0/1395729872 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc0d002ff60 msgr2=0x7fc0d0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:02.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:02.085+0000 7fc0ec42a700 1 --2- 172.21.15.194:0/1395729872 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc0d002ff60 0x7fc0d0032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fc0e00096a0 tx=0x7fc0e0006b40).stop 2024-01-18T06:52:02.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:02.085+0000 7fc0ec42a700 1 -- 172.21.15.194:0/1395729872 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc0e40f6510 msgr2=0x7fc0e4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:02.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:02.085+0000 7fc0ec42a700 1 --2- 172.21.15.194:0/1395729872 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc0e40f6510 0x7fc0e4106730 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fc0d4013510 tx=0x7fc0d400b040).stop 2024-01-18T06:52:02.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:02.085+0000 7fc0ec42a700 1 -- 172.21.15.194:0/1395729872 shutdown_connections 2024-01-18T06:52:02.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:02.085+0000 7fc0ec42a700 1 --2- 172.21.15.194:0/1395729872 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc0d002ff60 0x7fc0d0032420 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:02.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:02.085+0000 7fc0ec42a700 1 --2- 172.21.15.194:0/1395729872 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc0e40f6510 0x7fc0e4106730 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:02.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:02.085+0000 7fc0ec42a700 1 -- 172.21.15.194:0/1395729872 >> 172.21.15.194:0/1395729872 conn(0x7fc0e40f1c70 msgr2=0x7fc0e40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:52:02.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:02.085+0000 7fc0ec42a700 1 -- 172.21.15.194:0/1395729872 shutdown_connections 2024-01-18T06:52:02.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:02.085+0000 7fc0ec42a700 1 -- 172.21.15.194:0/1395729872 wait complete. 2024-01-18T06:52:02.086 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:52:03.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:03 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:52:02.006848+0000 mgr.smithi047.fuhzzr (mgr.14162) 107 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:52:03.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:03 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:52:02.084710+0000 mon.smithi047 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.194:0/1395729872' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:52:03.473 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:52:03.474 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:52:04.706 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:04.706+0000 7fbb6c225700 1 -- 172.21.15.194:0/3666829704 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbb640f6510 msgr2=0x7fbb640f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:04.706 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:04.706+0000 7fbb6c225700 1 --2- 172.21.15.194:0/3666829704 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbb640f6510 0x7fbb640f6930 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7fbb60009fb0 tx=0x7fbb60009b30).stop 2024-01-18T06:52:04.706 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:04.706+0000 7fbb6c225700 1 -- 172.21.15.194:0/3666829704 shutdown_connections 2024-01-18T06:52:04.706 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:04.706+0000 7fbb6c225700 1 --2- 172.21.15.194:0/3666829704 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbb640f6510 0x7fbb640f6930 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:04.707 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:04.706+0000 7fbb6c225700 1 -- 172.21.15.194:0/3666829704 >> 172.21.15.194:0/3666829704 conn(0x7fbb640f1c70 msgr2=0x7fbb640f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:52:04.707 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:04.706+0000 7fbb6c225700 1 -- 172.21.15.194:0/3666829704 shutdown_connections 2024-01-18T06:52:04.707 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:04.706+0000 7fbb6c225700 1 -- 172.21.15.194:0/3666829704 wait complete. 2024-01-18T06:52:04.707 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:04.707+0000 7fbb6c225700 1 Processor -- start 2024-01-18T06:52:04.707 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:04.707+0000 7fbb6c225700 1 -- start start 2024-01-18T06:52:04.707 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:04.707+0000 7fbb6c225700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbb640f6510 0x7fbb64106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:52:04.707 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:04.707+0000 7fbb6c225700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb64106c20 con 0x7fbb640f6510 2024-01-18T06:52:04.709 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:04.707+0000 7fbb69fc1700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbb640f6510 0x7fbb64106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:52:04.709 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:04.707+0000 7fbb69fc1700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbb640f6510 0x7fbb64106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:53858/0 (socket says 172.21.15.194:53858) 2024-01-18T06:52:04.709 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:04.707+0000 7fbb69fc1700 1 -- 172.21.15.194:0/1842486113 learned_addr learned my addr 172.21.15.194:0/1842486113 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:52:04.709 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:04.707+0000 7fbb69fc1700 1 -- 172.21.15.194:0/1842486113 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb60005040 con 0x7fbb640f6510 2024-01-18T06:52:04.709 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:04.708+0000 7fbb69fc1700 1 --2- 172.21.15.194:0/1842486113 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbb640f6510 0x7fbb64106730 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7fbb60013510 tx=0x7fbb6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:52:04.709 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:04.708+0000 7fbb5a7fc700 1 -- 172.21.15.194:0/1842486113 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb60009320 con 0x7fbb640f6510 2024-01-18T06:52:04.709 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:04.708+0000 7fbb6c225700 1 -- 172.21.15.194:0/1842486113 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb64106e20 con 0x7fbb640f6510 2024-01-18T06:52:04.709 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:04.708+0000 7fbb5a7fc700 1 -- 172.21.15.194:0/1842486113 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbb60013f00 con 0x7fbb640f6510 2024-01-18T06:52:04.709 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:04.708+0000 7fbb5a7fc700 1 -- 172.21.15.194:0/1842486113 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb60014d90 con 0x7fbb640f6510 2024-01-18T06:52:04.709 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:04.708+0000 7fbb6c225700 1 -- 172.21.15.194:0/1842486113 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb641072b0 con 0x7fbb640f6510 2024-01-18T06:52:04.709 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:04.709+0000 7fbb5a7fc700 1 -- 172.21.15.194:0/1842486113 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbb60014420 con 0x7fbb640f6510 2024-01-18T06:52:04.709 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:04.709+0000 7fbb5a7fc700 1 --2- 172.21.15.194:0/1842486113 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fbb5002ffb0 0x7fbb50032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:52:04.709 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:04.709+0000 7fbb5a7fc700 1 -- 172.21.15.194:0/1842486113 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbb60040fd0 con 0x7fbb640f6510 2024-01-18T06:52:04.710 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:04.709+0000 7fbb6c225700 1 -- 172.21.15.194:0/1842486113 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb64045ad0 con 0x7fbb640f6510 2024-01-18T06:52:04.710 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:04.709+0000 7fbb697c0700 1 --2- 172.21.15.194:0/1842486113 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fbb5002ffb0 0x7fbb50032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:52:04.710 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:04.709+0000 7fbb697c0700 1 --2- 172.21.15.194:0/1842486113 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fbb5002ffb0 0x7fbb50032470 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fbb540096a0 tx=0x7fbb54006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:52:04.713 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:04.714+0000 7fbb5a7fc700 1 -- 172.21.15.194:0/1842486113 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbb600095f0 con 0x7fbb640f6510 2024-01-18T06:52:04.919 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:04.919+0000 7fbb6c225700 1 -- 172.21.15.194:0/1842486113 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbb64107fd0 con 0x7fbb640f6510 2024-01-18T06:52:04.919 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:04.920+0000 7fbb5a7fc700 1 -- 172.21.15.194:0/1842486113 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbb6001db70 con 0x7fbb640f6510 2024-01-18T06:52:04.919 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:52:04.920 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:52:04.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:04.921+0000 7fbb6c225700 1 -- 172.21.15.194:0/1842486113 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fbb5002ffb0 msgr2=0x7fbb50032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:04.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:04.921+0000 7fbb6c225700 1 --2- 172.21.15.194:0/1842486113 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fbb5002ffb0 0x7fbb50032470 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fbb540096a0 tx=0x7fbb54006b40).stop 2024-01-18T06:52:04.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:04.921+0000 7fbb6c225700 1 -- 172.21.15.194:0/1842486113 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbb640f6510 msgr2=0x7fbb64106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:04.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:04.921+0000 7fbb6c225700 1 --2- 172.21.15.194:0/1842486113 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbb640f6510 0x7fbb64106730 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7fbb60013510 tx=0x7fbb6000b040).stop 2024-01-18T06:52:04.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:04.921+0000 7fbb6c225700 1 -- 172.21.15.194:0/1842486113 shutdown_connections 2024-01-18T06:52:04.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:04.922+0000 7fbb6c225700 1 --2- 172.21.15.194:0/1842486113 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fbb5002ffb0 0x7fbb50032470 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:04.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:04.922+0000 7fbb6c225700 1 --2- 172.21.15.194:0/1842486113 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbb640f6510 0x7fbb64106730 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:04.922 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:04.922+0000 7fbb6c225700 1 -- 172.21.15.194:0/1842486113 >> 172.21.15.194:0/1842486113 conn(0x7fbb640f1c70 msgr2=0x7fbb640f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:52:04.922 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:04.922+0000 7fbb6c225700 1 -- 172.21.15.194:0/1842486113 shutdown_connections 2024-01-18T06:52:04.922 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:04.922+0000 7fbb6c225700 1 -- 172.21.15.194:0/1842486113 wait complete. 2024-01-18T06:52:04.922 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:52:05.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:05 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:52:04.007078+0000 mgr.smithi047.fuhzzr (mgr.14162) 108 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:52:05.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:05 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:52:04.920838+0000 mon.smithi047 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.194:0/1842486113' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:52:06.317 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:52:06.318 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:52:07.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:07 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:52:06.007355+0000 mgr.smithi047.fuhzzr (mgr.14162) 109 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:52:07.549 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:07.547+0000 7f5e62bdc700 1 -- 172.21.15.194:0/3849518102 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e5c060200 msgr2=0x7f5e5c0fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:07.549 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:07.547+0000 7f5e62bdc700 1 --2- 172.21.15.194:0/3849518102 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e5c060200 0x7f5e5c0fb570 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f5e50009fb0 tx=0x7f5e50009b30).stop 2024-01-18T06:52:07.549 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:07.547+0000 7f5e62bdc700 1 -- 172.21.15.194:0/3849518102 shutdown_connections 2024-01-18T06:52:07.549 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:07.547+0000 7f5e62bdc700 1 --2- 172.21.15.194:0/3849518102 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e5c060200 0x7f5e5c0fb570 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:07.549 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:07.547+0000 7f5e62bdc700 1 -- 172.21.15.194:0/3849518102 >> 172.21.15.194:0/3849518102 conn(0x7f5e5c0f0b70 msgr2=0x7f5e5c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:52:07.549 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:07.547+0000 7f5e62bdc700 1 -- 172.21.15.194:0/3849518102 shutdown_connections 2024-01-18T06:52:07.549 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:07.547+0000 7f5e62bdc700 1 -- 172.21.15.194:0/3849518102 wait complete. 2024-01-18T06:52:07.549 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:07.548+0000 7f5e62bdc700 1 Processor -- start 2024-01-18T06:52:07.549 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:07.548+0000 7f5e62bdc700 1 -- start start 2024-01-18T06:52:07.550 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:07.548+0000 7f5e62bdc700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e5c060200 0x7f5e5c1088b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:52:07.550 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:07.548+0000 7f5e62bdc700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c108da0 con 0x7f5e5c060200 2024-01-18T06:52:07.550 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:07.548+0000 7f5e60978700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e5c060200 0x7f5e5c1088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:52:07.550 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:07.548+0000 7f5e60978700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e5c060200 0x7f5e5c1088b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:37234/0 (socket says 172.21.15.194:37234) 2024-01-18T06:52:07.550 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:07.548+0000 7f5e60978700 1 -- 172.21.15.194:0/3056390976 learned_addr learned my addr 172.21.15.194:0/3056390976 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:52:07.550 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:07.549+0000 7f5e60978700 1 -- 172.21.15.194:0/3056390976 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e50005040 con 0x7f5e5c060200 2024-01-18T06:52:07.550 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:07.549+0000 7f5e60978700 1 --2- 172.21.15.194:0/3056390976 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e5c060200 0x7f5e5c1088b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f5e50009830 tx=0x7f5e5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:52:07.550 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:07.549+0000 7f5e597fa700 1 -- 172.21.15.194:0/3056390976 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e50009320 con 0x7f5e5c060200 2024-01-18T06:52:07.550 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:07.549+0000 7f5e62bdc700 1 -- 172.21.15.194:0/3056390976 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e5c108fa0 con 0x7f5e5c060200 2024-01-18T06:52:07.550 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:07.549+0000 7f5e597fa700 1 -- 172.21.15.194:0/3056390976 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e500136c0 con 0x7f5e5c060200 2024-01-18T06:52:07.550 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:07.549+0000 7f5e597fa700 1 -- 172.21.15.194:0/3056390976 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e5001ee90 con 0x7f5e5c060200 2024-01-18T06:52:07.551 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:07.549+0000 7f5e62bdc700 1 -- 172.21.15.194:0/3056390976 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e5c109430 con 0x7f5e5c060200 2024-01-18T06:52:07.551 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:07.550+0000 7f5e597fa700 1 -- 172.21.15.194:0/3056390976 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5e5001e420 con 0x7f5e5c060200 2024-01-18T06:52:07.551 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:07.550+0000 7f5e62bdc700 1 -- 172.21.15.194:0/3056390976 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e5c045ad0 con 0x7f5e5c060200 2024-01-18T06:52:07.551 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:07.550+0000 7f5e597fa700 1 --2- 172.21.15.194:0/3056390976 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f5e4c02ff10 0x7f5e4c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:52:07.551 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:07.550+0000 7f5e597fa700 1 -- 172.21.15.194:0/3056390976 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5e50040f10 con 0x7f5e5c060200 2024-01-18T06:52:07.551 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:07.550+0000 7f5e5bfff700 1 --2- 172.21.15.194:0/3056390976 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f5e4c02ff10 0x7f5e4c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:52:07.552 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:07.551+0000 7f5e5bfff700 1 --2- 172.21.15.194:0/3056390976 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f5e4c02ff10 0x7f5e4c0323d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f5e480096a0 tx=0x7f5e48006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:52:07.554 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:07.555+0000 7f5e597fa700 1 -- 172.21.15.194:0/3056390976 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5e500095f0 con 0x7f5e5c060200 2024-01-18T06:52:07.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:07.758+0000 7f5e62bdc700 1 -- 172.21.15.194:0/3056390976 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5e5c10a2e0 con 0x7f5e5c060200 2024-01-18T06:52:07.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:07.759+0000 7f5e597fa700 1 -- 172.21.15.194:0/3056390976 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5e5002c2d0 con 0x7f5e5c060200 2024-01-18T06:52:07.758 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:52:07.758 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:52:07.760 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:07.760+0000 7f5e62bdc700 1 -- 172.21.15.194:0/3056390976 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f5e4c02ff10 msgr2=0x7f5e4c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:07.760 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:07.760+0000 7f5e62bdc700 1 --2- 172.21.15.194:0/3056390976 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f5e4c02ff10 0x7f5e4c0323d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f5e480096a0 tx=0x7f5e48006b40).stop 2024-01-18T06:52:07.760 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:07.760+0000 7f5e62bdc700 1 -- 172.21.15.194:0/3056390976 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e5c060200 msgr2=0x7f5e5c1088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:07.760 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:07.760+0000 7f5e62bdc700 1 --2- 172.21.15.194:0/3056390976 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e5c060200 0x7f5e5c1088b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f5e50009830 tx=0x7f5e5000b040).stop 2024-01-18T06:52:07.760 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:07.760+0000 7f5e62bdc700 1 -- 172.21.15.194:0/3056390976 shutdown_connections 2024-01-18T06:52:07.761 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:07.760+0000 7f5e62bdc700 1 --2- 172.21.15.194:0/3056390976 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f5e4c02ff10 0x7f5e4c0323d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:07.761 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:07.760+0000 7f5e62bdc700 1 --2- 172.21.15.194:0/3056390976 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e5c060200 0x7f5e5c1088b0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:07.761 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:07.760+0000 7f5e62bdc700 1 -- 172.21.15.194:0/3056390976 >> 172.21.15.194:0/3056390976 conn(0x7f5e5c0f0b70 msgr2=0x7f5e5c0f1650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:52:07.761 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:07.761+0000 7f5e62bdc700 1 -- 172.21.15.194:0/3056390976 shutdown_connections 2024-01-18T06:52:07.761 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:07.761+0000 7f5e62bdc700 1 -- 172.21.15.194:0/3056390976 wait complete. 2024-01-18T06:52:07.761 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:52:08.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:08 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:52:07.759749+0000 mon.smithi047 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.194:0/3056390976' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:52:09.157 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:52:09.157 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:52:09.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:09 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:52:08.007648+0000 mgr.smithi047.fuhzzr (mgr.14162) 110 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:52:10.329 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:10.328+0000 7fddc434a700 1 -- 172.21.15.194:0/748832896 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fddbc0f5410 msgr2=0x7fddbc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:10.329 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:10.328+0000 7fddc434a700 1 --2- 172.21.15.194:0/748832896 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fddbc0f5410 0x7fddbc0f5830 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7fddac009fb0 tx=0x7fddac009b30).stop 2024-01-18T06:52:10.329 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:10.328+0000 7fddc434a700 1 -- 172.21.15.194:0/748832896 shutdown_connections 2024-01-18T06:52:10.329 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:10.328+0000 7fddc434a700 1 --2- 172.21.15.194:0/748832896 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fddbc0f5410 0x7fddbc0f5830 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:10.329 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:10.328+0000 7fddc434a700 1 -- 172.21.15.194:0/748832896 >> 172.21.15.194:0/748832896 conn(0x7fddbc0f0b70 msgr2=0x7fddbc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:52:10.329 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:10.328+0000 7fddc434a700 1 -- 172.21.15.194:0/748832896 shutdown_connections 2024-01-18T06:52:10.329 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:10.328+0000 7fddc434a700 1 -- 172.21.15.194:0/748832896 wait complete. 2024-01-18T06:52:10.329 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:10.328+0000 7fddc434a700 1 Processor -- start 2024-01-18T06:52:10.329 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:10.328+0000 7fddc434a700 1 -- start start 2024-01-18T06:52:10.330 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:10.328+0000 7fddc434a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fddbc0f5410 0x7fddbc0fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:52:10.330 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:10.328+0000 7fddc434a700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddbc0fb820 con 0x7fddbc0f5410 2024-01-18T06:52:10.330 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:10.329+0000 7fddc20e6700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fddbc0f5410 0x7fddbc0fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:52:10.330 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:10.329+0000 7fddc20e6700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fddbc0f5410 0x7fddbc0fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:37258/0 (socket says 172.21.15.194:37258) 2024-01-18T06:52:10.330 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:10.329+0000 7fddc20e6700 1 -- 172.21.15.194:0/1925679701 learned_addr learned my addr 172.21.15.194:0/1925679701 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:52:10.330 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:10.329+0000 7fddc20e6700 1 -- 172.21.15.194:0/1925679701 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddac005040 con 0x7fddbc0f5410 2024-01-18T06:52:10.330 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:10.329+0000 7fddc20e6700 1 --2- 172.21.15.194:0/1925679701 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fddbc0f5410 0x7fddbc0fd3a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fddac013510 tx=0x7fddac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:52:10.330 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:10.329+0000 7fddb2ffd700 1 -- 172.21.15.194:0/1925679701 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fddac009320 con 0x7fddbc0f5410 2024-01-18T06:52:10.330 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:10.329+0000 7fddb2ffd700 1 -- 172.21.15.194:0/1925679701 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fddac013f00 con 0x7fddbc0f5410 2024-01-18T06:52:10.330 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:10.329+0000 7fddc434a700 1 -- 172.21.15.194:0/1925679701 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddbc0fb9c0 con 0x7fddbc0f5410 2024-01-18T06:52:10.330 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:10.329+0000 7fddb2ffd700 1 -- 172.21.15.194:0/1925679701 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fddac014d90 con 0x7fddbc0f5410 2024-01-18T06:52:10.330 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:10.329+0000 7fddc434a700 1 -- 172.21.15.194:0/1925679701 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddbc0fbe50 con 0x7fddbc0f5410 2024-01-18T06:52:10.331 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:10.330+0000 7fddb2ffd700 1 -- 172.21.15.194:0/1925679701 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fddac0095f0 con 0x7fddbc0f5410 2024-01-18T06:52:10.331 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:10.330+0000 7fddc434a700 1 -- 172.21.15.194:0/1925679701 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddbc059100 con 0x7fddbc0f5410 2024-01-18T06:52:10.331 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:10.330+0000 7fddb2ffd700 1 --2- 172.21.15.194:0/1925679701 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fdda802ffb0 0x7fdda8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:52:10.331 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:10.330+0000 7fddb2ffd700 1 -- 172.21.15.194:0/1925679701 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fddac0410c0 con 0x7fddbc0f5410 2024-01-18T06:52:10.331 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:10.331+0000 7fddc18e5700 1 --2- 172.21.15.194:0/1925679701 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fdda802ffb0 0x7fdda8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:52:10.331 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:10.331+0000 7fddc18e5700 1 --2- 172.21.15.194:0/1925679701 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fdda802ffb0 0x7fdda8032470 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fddb80096a0 tx=0x7fddb8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:52:10.334 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:10.335+0000 7fddb2ffd700 1 -- 172.21.15.194:0/1925679701 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fddac014420 con 0x7fddbc0f5410 2024-01-18T06:52:10.542 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:10.543+0000 7fddc434a700 1 -- 172.21.15.194:0/1925679701 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fddbc0fae60 con 0x7fddbc0f5410 2024-01-18T06:52:10.543 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:10.543+0000 7fddb2ffd700 1 -- 172.21.15.194:0/1925679701 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fddac010230 con 0x7fddbc0f5410 2024-01-18T06:52:10.543 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:52:10.543 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:52:10.545 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:10.545+0000 7fddc434a700 1 -- 172.21.15.194:0/1925679701 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fdda802ffb0 msgr2=0x7fdda8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:10.545 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:10.545+0000 7fddc434a700 1 --2- 172.21.15.194:0/1925679701 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fdda802ffb0 0x7fdda8032470 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fddb80096a0 tx=0x7fddb8006b40).stop 2024-01-18T06:52:10.545 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:10.545+0000 7fddc434a700 1 -- 172.21.15.194:0/1925679701 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fddbc0f5410 msgr2=0x7fddbc0fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:10.545 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:10.545+0000 7fddc434a700 1 --2- 172.21.15.194:0/1925679701 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fddbc0f5410 0x7fddbc0fd3a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fddac013510 tx=0x7fddac00b040).stop 2024-01-18T06:52:10.545 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:10.545+0000 7fddc434a700 1 -- 172.21.15.194:0/1925679701 shutdown_connections 2024-01-18T06:52:10.545 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:10.545+0000 7fddc434a700 1 --2- 172.21.15.194:0/1925679701 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fdda802ffb0 0x7fdda8032470 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:10.545 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:10.545+0000 7fddc434a700 1 --2- 172.21.15.194:0/1925679701 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fddbc0f5410 0x7fddbc0fd3a0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:10.545 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:10.545+0000 7fddc434a700 1 -- 172.21.15.194:0/1925679701 >> 172.21.15.194:0/1925679701 conn(0x7fddbc0f0b70 msgr2=0x7fddbc05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:52:10.545 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:10.545+0000 7fddc434a700 1 -- 172.21.15.194:0/1925679701 shutdown_connections 2024-01-18T06:52:10.545 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:10.545+0000 7fddc434a700 1 -- 172.21.15.194:0/1925679701 wait complete. 2024-01-18T06:52:10.545 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:52:11.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:11 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:52:10.007967+0000 mgr.smithi047.fuhzzr (mgr.14162) 111 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:52:11.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:11 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:52:10.544240+0000 mon.smithi047 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.194:0/1925679701' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:52:11.937 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:52:11.937 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:52:13.125 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:13.120+0000 7f4ad7908700 1 -- 172.21.15.194:0/3102564752 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ad00f5410 msgr2=0x7f4ad00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:13.125 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:13.120+0000 7f4ad7908700 1 --2- 172.21.15.194:0/3102564752 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ad00f5410 0x7f4ad00f5830 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f4ac0009fb0 tx=0x7f4ac0009b30).stop 2024-01-18T06:52:13.126 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:13.121+0000 7f4ad7908700 1 -- 172.21.15.194:0/3102564752 shutdown_connections 2024-01-18T06:52:13.126 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:13.121+0000 7f4ad7908700 1 --2- 172.21.15.194:0/3102564752 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ad00f5410 0x7f4ad00f5830 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:13.126 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:13.121+0000 7f4ad7908700 1 -- 172.21.15.194:0/3102564752 >> 172.21.15.194:0/3102564752 conn(0x7f4ad00f0b70 msgr2=0x7f4ad00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:52:13.126 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:13.121+0000 7f4ad7908700 1 -- 172.21.15.194:0/3102564752 shutdown_connections 2024-01-18T06:52:13.126 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:13.121+0000 7f4ad7908700 1 -- 172.21.15.194:0/3102564752 wait complete. 2024-01-18T06:52:13.126 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:13.121+0000 7f4ad7908700 1 Processor -- start 2024-01-18T06:52:13.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:13.121+0000 7f4ad7908700 1 -- start start 2024-01-18T06:52:13.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:13.121+0000 7f4ad7908700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ad00f5410 0x7f4ad00fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:52:13.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:13.121+0000 7f4ad7908700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad00fb820 con 0x7f4ad00f5410 2024-01-18T06:52:13.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:13.121+0000 7f4ad56a4700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ad00f5410 0x7f4ad00fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:52:13.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:13.121+0000 7f4ad56a4700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ad00f5410 0x7f4ad00fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:37272/0 (socket says 172.21.15.194:37272) 2024-01-18T06:52:13.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:13.121+0000 7f4ad56a4700 1 -- 172.21.15.194:0/3767233541 learned_addr learned my addr 172.21.15.194:0/3767233541 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:52:13.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:13.122+0000 7f4ad56a4700 1 -- 172.21.15.194:0/3767233541 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ac0005040 con 0x7f4ad00f5410 2024-01-18T06:52:13.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:13.122+0000 7f4ad56a4700 1 --2- 172.21.15.194:0/3767233541 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ad00f5410 0x7f4ad00fd3a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f4ac0013510 tx=0x7f4ac000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:52:13.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:13.122+0000 7f4ac5ffb700 1 -- 172.21.15.194:0/3767233541 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ac0009320 con 0x7f4ad00f5410 2024-01-18T06:52:13.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:13.122+0000 7f4ad7908700 1 -- 172.21.15.194:0/3767233541 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ad00fb9c0 con 0x7f4ad00f5410 2024-01-18T06:52:13.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:13.122+0000 7f4ac5ffb700 1 -- 172.21.15.194:0/3767233541 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4ac0013f00 con 0x7f4ad00f5410 2024-01-18T06:52:13.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:13.122+0000 7f4ac5ffb700 1 -- 172.21.15.194:0/3767233541 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ac0014d90 con 0x7f4ad00f5410 2024-01-18T06:52:13.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:13.122+0000 7f4ad7908700 1 -- 172.21.15.194:0/3767233541 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ad00fbe50 con 0x7f4ad00f5410 2024-01-18T06:52:13.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:13.123+0000 7f4ac5ffb700 1 -- 172.21.15.194:0/3767233541 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4ac00095f0 con 0x7f4ad00f5410 2024-01-18T06:52:13.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:13.123+0000 7f4ad7908700 1 -- 172.21.15.194:0/3767233541 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ad0059100 con 0x7f4ad00f5410 2024-01-18T06:52:13.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:13.123+0000 7f4ac5ffb700 1 --2- 172.21.15.194:0/3767233541 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f4abc02ff60 0x7f4abc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:52:13.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:13.123+0000 7f4ac5ffb700 1 -- 172.21.15.194:0/3767233541 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4ac0041080 con 0x7f4ad00f5410 2024-01-18T06:52:13.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:13.123+0000 7f4ad4ea3700 1 --2- 172.21.15.194:0/3767233541 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f4abc02ff60 0x7f4abc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:52:13.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:13.124+0000 7f4ad4ea3700 1 --2- 172.21.15.194:0/3767233541 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f4abc02ff60 0x7f4abc032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f4acc0096a0 tx=0x7f4acc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:52:13.129 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:13.128+0000 7f4ac5ffb700 1 -- 172.21.15.194:0/3767233541 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4ac0014420 con 0x7f4ad00f5410 2024-01-18T06:52:13.332 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:13.332+0000 7f4ad7908700 1 -- 172.21.15.194:0/3767233541 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4ad00fae60 con 0x7f4ad00f5410 2024-01-18T06:52:13.333 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:13.333+0000 7f4ac5ffb700 1 -- 172.21.15.194:0/3767233541 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4ac001d7f0 con 0x7f4ad00f5410 2024-01-18T06:52:13.333 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:52:13.333 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:52:13.335 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:13.334+0000 7f4ad7908700 1 -- 172.21.15.194:0/3767233541 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f4abc02ff60 msgr2=0x7f4abc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:13.335 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:13.334+0000 7f4ad7908700 1 --2- 172.21.15.194:0/3767233541 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f4abc02ff60 0x7f4abc032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f4acc0096a0 tx=0x7f4acc006b40).stop 2024-01-18T06:52:13.335 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:13.334+0000 7f4ad7908700 1 -- 172.21.15.194:0/3767233541 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ad00f5410 msgr2=0x7f4ad00fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:13.336 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:13.334+0000 7f4ad7908700 1 --2- 172.21.15.194:0/3767233541 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ad00f5410 0x7f4ad00fd3a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f4ac0013510 tx=0x7f4ac000b040).stop 2024-01-18T06:52:13.336 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:13.334+0000 7f4ad7908700 1 -- 172.21.15.194:0/3767233541 shutdown_connections 2024-01-18T06:52:13.336 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:13.334+0000 7f4ad7908700 1 --2- 172.21.15.194:0/3767233541 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f4abc02ff60 0x7f4abc032420 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:13.336 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:13.334+0000 7f4ad7908700 1 --2- 172.21.15.194:0/3767233541 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ad00f5410 0x7f4ad00fd3a0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:13.336 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:13.334+0000 7f4ad7908700 1 -- 172.21.15.194:0/3767233541 >> 172.21.15.194:0/3767233541 conn(0x7f4ad00f0b70 msgr2=0x7f4ad005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:52:13.336 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:13.334+0000 7f4ad7908700 1 -- 172.21.15.194:0/3767233541 shutdown_connections 2024-01-18T06:52:13.336 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:13.334+0000 7f4ad7908700 1 -- 172.21.15.194:0/3767233541 wait complete. 2024-01-18T06:52:13.336 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:52:13.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:13 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:52:12.008303+0000 mgr.smithi047.fuhzzr (mgr.14162) 112 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:52:14.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:14 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:52:13.334594+0000 mon.smithi047 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.194:0/3767233541' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:52:14.722 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:52:14.722 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:52:15.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:15 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:52:14.008602+0000 mgr.smithi047.fuhzzr (mgr.14162) 113 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:52:15.938 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:15.937+0000 7fa48dd7d700 1 -- 172.21.15.194:0/2271345744 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4880f6510 msgr2=0x7fa4880f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:15.938 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:15.937+0000 7fa48dd7d700 1 --2- 172.21.15.194:0/2271345744 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4880f6510 0x7fa4880f6930 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7fa480009960 tx=0x7fa480004d10).stop 2024-01-18T06:52:15.938 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:15.937+0000 7fa48dd7d700 1 -- 172.21.15.194:0/2271345744 shutdown_connections 2024-01-18T06:52:15.938 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:15.937+0000 7fa48dd7d700 1 --2- 172.21.15.194:0/2271345744 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4880f6510 0x7fa4880f6930 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:15.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:15.937+0000 7fa48dd7d700 1 -- 172.21.15.194:0/2271345744 >> 172.21.15.194:0/2271345744 conn(0x7fa4880f1c70 msgr2=0x7fa4880f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:52:15.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:15.937+0000 7fa48dd7d700 1 -- 172.21.15.194:0/2271345744 shutdown_connections 2024-01-18T06:52:15.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:15.937+0000 7fa48dd7d700 1 -- 172.21.15.194:0/2271345744 wait complete. 2024-01-18T06:52:15.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:15.937+0000 7fa48dd7d700 1 Processor -- start 2024-01-18T06:52:15.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:15.938+0000 7fa48dd7d700 1 -- start start 2024-01-18T06:52:15.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:15.938+0000 7fa48dd7d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4880f6510 0x7fa488106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:52:15.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:15.938+0000 7fa48dd7d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa488106c20 con 0x7fa4880f6510 2024-01-18T06:52:15.941 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:15.938+0000 7fa4877fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4880f6510 0x7fa488106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:52:15.941 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:15.938+0000 7fa4877fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4880f6510 0x7fa488106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:37284/0 (socket says 172.21.15.194:37284) 2024-01-18T06:52:15.941 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:15.938+0000 7fa4877fe700 1 -- 172.21.15.194:0/2519500097 learned_addr learned my addr 172.21.15.194:0/2519500097 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:52:15.941 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:15.938+0000 7fa4877fe700 1 -- 172.21.15.194:0/2519500097 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa480009640 con 0x7fa4880f6510 2024-01-18T06:52:15.941 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:15.938+0000 7fa4877fe700 1 --2- 172.21.15.194:0/2519500097 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4880f6510 0x7fa488106730 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fa480004b70 tx=0x7fa480004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:52:15.941 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:15.939+0000 7fa46ffff700 1 -- 172.21.15.194:0/2519500097 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa480018070 con 0x7fa4880f6510 2024-01-18T06:52:15.941 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:15.939+0000 7fa48dd7d700 1 -- 172.21.15.194:0/2519500097 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa488106e20 con 0x7fa4880f6510 2024-01-18T06:52:15.941 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:15.939+0000 7fa46ffff700 1 -- 172.21.15.194:0/2519500097 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa48000b870 con 0x7fa4880f6510 2024-01-18T06:52:15.941 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:15.939+0000 7fa46ffff700 1 -- 172.21.15.194:0/2519500097 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa48001c560 con 0x7fa4880f6510 2024-01-18T06:52:15.941 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:15.939+0000 7fa48dd7d700 1 -- 172.21.15.194:0/2519500097 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4881072b0 con 0x7fa4880f6510 2024-01-18T06:52:15.941 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:15.940+0000 7fa46ffff700 1 -- 172.21.15.194:0/2519500097 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa48001c6c0 con 0x7fa4880f6510 2024-01-18T06:52:15.942 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:15.940+0000 7fa46ffff700 1 --2- 172.21.15.194:0/2519500097 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa470030000 0x7fa4700324c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:52:15.942 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:15.940+0000 7fa46ffff700 1 -- 172.21.15.194:0/2519500097 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa480041280 con 0x7fa4880f6510 2024-01-18T06:52:15.942 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:15.940+0000 7fa48dd7d700 1 -- 172.21.15.194:0/2519500097 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa488045ad0 con 0x7fa4880f6510 2024-01-18T06:52:15.942 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:15.940+0000 7fa486ffd700 1 --2- 172.21.15.194:0/2519500097 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa470030000 0x7fa4700324c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:52:15.942 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:15.940+0000 7fa486ffd700 1 --2- 172.21.15.194:0/2519500097 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa470030000 0x7fa4700324c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fa47c0096a0 tx=0x7fa47c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:52:15.945 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:15.944+0000 7fa46ffff700 1 -- 172.21.15.194:0/2519500097 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa480012bc0 con 0x7fa4880f6510 2024-01-18T06:52:16.154 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:16.154+0000 7fa48dd7d700 1 -- 172.21.15.194:0/2519500097 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa4881082b0 con 0x7fa4880f6510 2024-01-18T06:52:16.155 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:16.155+0000 7fa46ffff700 1 -- 172.21.15.194:0/2519500097 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa480012990 con 0x7fa4880f6510 2024-01-18T06:52:16.155 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:52:16.155 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:52:16.157 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:16.156+0000 7fa48dd7d700 1 -- 172.21.15.194:0/2519500097 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa470030000 msgr2=0x7fa4700324c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:16.157 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:16.156+0000 7fa48dd7d700 1 --2- 172.21.15.194:0/2519500097 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa470030000 0x7fa4700324c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fa47c0096a0 tx=0x7fa47c006b40).stop 2024-01-18T06:52:16.157 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:16.156+0000 7fa48dd7d700 1 -- 172.21.15.194:0/2519500097 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4880f6510 msgr2=0x7fa488106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:16.157 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:16.156+0000 7fa48dd7d700 1 --2- 172.21.15.194:0/2519500097 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4880f6510 0x7fa488106730 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fa480004b70 tx=0x7fa480004d10).stop 2024-01-18T06:52:16.157 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:16.156+0000 7fa48dd7d700 1 -- 172.21.15.194:0/2519500097 shutdown_connections 2024-01-18T06:52:16.157 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:16.156+0000 7fa48dd7d700 1 --2- 172.21.15.194:0/2519500097 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa470030000 0x7fa4700324c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:16.157 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:16.156+0000 7fa48dd7d700 1 --2- 172.21.15.194:0/2519500097 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4880f6510 0x7fa488106730 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:16.157 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:16.156+0000 7fa48dd7d700 1 -- 172.21.15.194:0/2519500097 >> 172.21.15.194:0/2519500097 conn(0x7fa4880f1c70 msgr2=0x7fa4880f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:52:16.158 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:16.156+0000 7fa48dd7d700 1 -- 172.21.15.194:0/2519500097 shutdown_connections 2024-01-18T06:52:16.158 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:16.156+0000 7fa48dd7d700 1 -- 172.21.15.194:0/2519500097 wait complete. 2024-01-18T06:52:16.158 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:52:17.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:17 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:52:16.008891+0000 mgr.smithi047.fuhzzr (mgr.14162) 114 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:52:17.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:17 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:52:16.156605+0000 mon.smithi047 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.194:0/2519500097' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:52:17.559 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:52:17.559 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:52:18.770 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:18.770+0000 7f99c5afa700 1 -- 172.21.15.194:0/2574797613 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f99c00f51a0 msgr2=0x7f99c00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:18.770 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:18.770+0000 7f99c5afa700 1 --2- 172.21.15.194:0/2574797613 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f99c00f51a0 0x7f99c00f55c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f99b4009fb0 tx=0x7f99b4009b30).stop 2024-01-18T06:52:18.770 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:18.770+0000 7f99c5afa700 1 -- 172.21.15.194:0/2574797613 shutdown_connections 2024-01-18T06:52:18.771 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:18.770+0000 7f99c5afa700 1 --2- 172.21.15.194:0/2574797613 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f99c00f51a0 0x7f99c00f55c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:18.771 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:18.770+0000 7f99c5afa700 1 -- 172.21.15.194:0/2574797613 >> 172.21.15.194:0/2574797613 conn(0x7f99c00f09a0 msgr2=0x7f99c00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:52:18.771 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:18.770+0000 7f99c5afa700 1 -- 172.21.15.194:0/2574797613 shutdown_connections 2024-01-18T06:52:18.771 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:18.770+0000 7f99c5afa700 1 -- 172.21.15.194:0/2574797613 wait complete. 2024-01-18T06:52:18.773 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:18.770+0000 7f99c5afa700 1 Processor -- start 2024-01-18T06:52:18.773 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:18.770+0000 7f99c5afa700 1 -- start start 2024-01-18T06:52:18.773 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:18.771+0000 7f99c5afa700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f99c00f51a0 0x7f99c0109170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:52:18.773 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:18.771+0000 7f99c5afa700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c0060280 con 0x7f99c00f51a0 2024-01-18T06:52:18.773 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:18.771+0000 7f99c4af8700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f99c00f51a0 0x7f99c0109170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:52:18.773 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:18.771+0000 7f99c4af8700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f99c00f51a0 0x7f99c0109170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:58892/0 (socket says 172.21.15.194:58892) 2024-01-18T06:52:18.773 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:18.771+0000 7f99c4af8700 1 -- 172.21.15.194:0/3631505469 learned_addr learned my addr 172.21.15.194:0/3631505469 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:52:18.773 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:18.772+0000 7f99c4af8700 1 -- 172.21.15.194:0/3631505469 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b4005040 con 0x7f99c00f51a0 2024-01-18T06:52:18.773 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:18.772+0000 7f99c4af8700 1 --2- 172.21.15.194:0/3631505469 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f99c00f51a0 0x7f99c0109170 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f99b4013510 tx=0x7f99b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:52:18.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:18.772+0000 7f99bd7fa700 1 -- 172.21.15.194:0/3631505469 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99b4009320 con 0x7f99c00f51a0 2024-01-18T06:52:18.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:18.772+0000 7f99c5afa700 1 -- 172.21.15.194:0/3631505469 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99c0060480 con 0x7f99c00f51a0 2024-01-18T06:52:18.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:18.772+0000 7f99bd7fa700 1 -- 172.21.15.194:0/3631505469 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f99b4013f00 con 0x7f99c00f51a0 2024-01-18T06:52:18.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:18.772+0000 7f99bd7fa700 1 -- 172.21.15.194:0/3631505469 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99b4014dc0 con 0x7f99c00f51a0 2024-01-18T06:52:18.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:18.772+0000 7f99c5afa700 1 -- 172.21.15.194:0/3631505469 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99c01098a0 con 0x7f99c00f51a0 2024-01-18T06:52:18.775 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:18.773+0000 7f99bd7fa700 1 -- 172.21.15.194:0/3631505469 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f99b4014420 con 0x7f99c00f51a0 2024-01-18T06:52:18.775 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:18.773+0000 7f99c5afa700 1 -- 172.21.15.194:0/3631505469 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99c0045ad0 con 0x7f99c00f51a0 2024-01-18T06:52:18.775 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:18.773+0000 7f99bd7fa700 1 --2- 172.21.15.194:0/3631505469 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f99b002ffb0 0x7f99b0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:52:18.775 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:18.773+0000 7f99bd7fa700 1 -- 172.21.15.194:0/3631505469 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f99b4041090 con 0x7f99c00f51a0 2024-01-18T06:52:18.775 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:18.773+0000 7f99bffff700 1 --2- 172.21.15.194:0/3631505469 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f99b002ffb0 0x7f99b0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:52:18.775 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:18.773+0000 7f99bffff700 1 --2- 172.21.15.194:0/3631505469 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f99b002ffb0 0x7f99b0032470 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f99ac0096a0 tx=0x7f99ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:52:18.778 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:18.777+0000 7f99bd7fa700 1 -- 172.21.15.194:0/3631505469 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f99b401db70 con 0x7f99c00f51a0 2024-01-18T06:52:18.982 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:18.982+0000 7f99c5afa700 1 -- 172.21.15.194:0/3631505469 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f99c00fabe0 con 0x7f99c00f51a0 2024-01-18T06:52:18.983 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:18.982+0000 7f99bd7fa700 1 -- 172.21.15.194:0/3631505469 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f99b401db70 con 0x7f99c00f51a0 2024-01-18T06:52:18.983 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:52:18.983 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:52:18.984 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:18.984+0000 7f99c5afa700 1 -- 172.21.15.194:0/3631505469 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f99b002ffb0 msgr2=0x7f99b0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:18.984 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:18.984+0000 7f99c5afa700 1 --2- 172.21.15.194:0/3631505469 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f99b002ffb0 0x7f99b0032470 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f99ac0096a0 tx=0x7f99ac006b40).stop 2024-01-18T06:52:18.984 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:18.984+0000 7f99c5afa700 1 -- 172.21.15.194:0/3631505469 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f99c00f51a0 msgr2=0x7f99c0109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:18.985 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:18.984+0000 7f99c5afa700 1 --2- 172.21.15.194:0/3631505469 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f99c00f51a0 0x7f99c0109170 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f99b4013510 tx=0x7f99b400b040).stop 2024-01-18T06:52:18.985 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:18.984+0000 7f99c5afa700 1 -- 172.21.15.194:0/3631505469 shutdown_connections 2024-01-18T06:52:18.985 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:18.984+0000 7f99c5afa700 1 --2- 172.21.15.194:0/3631505469 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f99b002ffb0 0x7f99b0032470 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:18.985 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:18.984+0000 7f99c5afa700 1 --2- 172.21.15.194:0/3631505469 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f99c00f51a0 0x7f99c0109170 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:18.985 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:18.984+0000 7f99c5afa700 1 -- 172.21.15.194:0/3631505469 >> 172.21.15.194:0/3631505469 conn(0x7f99c00f09a0 msgr2=0x7f99c00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:52:18.985 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:18.984+0000 7f99c5afa700 1 -- 172.21.15.194:0/3631505469 shutdown_connections 2024-01-18T06:52:18.985 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:18.984+0000 7f99c5afa700 1 -- 172.21.15.194:0/3631505469 wait complete. 2024-01-18T06:52:18.985 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:52:19.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:18 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:52:18.009225+0000 mgr.smithi047.fuhzzr (mgr.14162) 115 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:52:20.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:19 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:52:18.984269+0000 mon.smithi047 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.194:0/3631505469' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:52:20.370 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:52:20.370 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:52:21.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:20 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:52:20.009611+0000 mgr.smithi047.fuhzzr (mgr.14162) 116 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:52:21.587 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:21.587+0000 7f95197d8700 1 -- 172.21.15.194:0/300056065 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f95140f51a0 msgr2=0x7f95140f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:21.588 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:21.587+0000 7f95197d8700 1 --2- 172.21.15.194:0/300056065 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f95140f51a0 0x7f95140f55c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f9504009fb0 tx=0x7f9504009b30).stop 2024-01-18T06:52:21.588 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:21.587+0000 7f95197d8700 1 -- 172.21.15.194:0/300056065 shutdown_connections 2024-01-18T06:52:21.588 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:21.587+0000 7f95197d8700 1 --2- 172.21.15.194:0/300056065 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f95140f51a0 0x7f95140f55c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:21.588 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:21.587+0000 7f95197d8700 1 -- 172.21.15.194:0/300056065 >> 172.21.15.194:0/300056065 conn(0x7f95140f09a0 msgr2=0x7f95140f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:52:21.588 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:21.587+0000 7f95197d8700 1 -- 172.21.15.194:0/300056065 shutdown_connections 2024-01-18T06:52:21.588 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:21.587+0000 7f95197d8700 1 -- 172.21.15.194:0/300056065 wait complete. 2024-01-18T06:52:21.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:21.587+0000 7f95197d8700 1 Processor -- start 2024-01-18T06:52:21.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:21.588+0000 7f95197d8700 1 -- start start 2024-01-18T06:52:21.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:21.588+0000 7f95197d8700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f95140f51a0 0x7f95141064c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:52:21.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:21.588+0000 7f95197d8700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95141069b0 con 0x7f95140f51a0 2024-01-18T06:52:21.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:21.588+0000 7f9513fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f95140f51a0 0x7f95141064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:52:21.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:21.588+0000 7f9513fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f95140f51a0 0x7f95141064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:58908/0 (socket says 172.21.15.194:58908) 2024-01-18T06:52:21.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:21.588+0000 7f9513fff700 1 -- 172.21.15.194:0/2394112729 learned_addr learned my addr 172.21.15.194:0/2394112729 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:52:21.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:21.589+0000 7f9513fff700 1 -- 172.21.15.194:0/2394112729 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9504005040 con 0x7f95140f51a0 2024-01-18T06:52:21.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:21.589+0000 7f9513fff700 1 --2- 172.21.15.194:0/2394112729 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f95140f51a0 0x7f95141064c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f9504013510 tx=0x7f950400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:52:21.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:21.589+0000 7f9510ff9700 1 -- 172.21.15.194:0/2394112729 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9504009320 con 0x7f95140f51a0 2024-01-18T06:52:21.591 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:21.589+0000 7f95197d8700 1 -- 172.21.15.194:0/2394112729 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9514106bb0 con 0x7f95140f51a0 2024-01-18T06:52:21.591 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:21.589+0000 7f9510ff9700 1 -- 172.21.15.194:0/2394112729 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9504013f00 con 0x7f95140f51a0 2024-01-18T06:52:21.591 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:21.589+0000 7f9510ff9700 1 -- 172.21.15.194:0/2394112729 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9504014dc0 con 0x7f95140f51a0 2024-01-18T06:52:21.591 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:21.589+0000 7f95197d8700 1 -- 172.21.15.194:0/2394112729 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9514107040 con 0x7f95140f51a0 2024-01-18T06:52:21.591 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:21.590+0000 7f9510ff9700 1 -- 172.21.15.194:0/2394112729 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f95040263a0 con 0x7f95140f51a0 2024-01-18T06:52:21.592 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:21.590+0000 7f9510ff9700 1 --2- 172.21.15.194:0/2394112729 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f94fc02ffb0 0x7f94fc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:52:21.592 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:21.590+0000 7f95197d8700 1 -- 172.21.15.194:0/2394112729 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9514045ad0 con 0x7f95140f51a0 2024-01-18T06:52:21.592 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:21.590+0000 7f9510ff9700 1 -- 172.21.15.194:0/2394112729 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9504041000 con 0x7f95140f51a0 2024-01-18T06:52:21.592 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:21.590+0000 7f95137fe700 1 --2- 172.21.15.194:0/2394112729 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f94fc02ffb0 0x7f94fc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:52:21.592 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:21.591+0000 7f95137fe700 1 --2- 172.21.15.194:0/2394112729 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f94fc02ffb0 0x7f94fc032470 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f95080096a0 tx=0x7f9508006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:52:21.595 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:21.596+0000 7f9510ff9700 1 -- 172.21.15.194:0/2394112729 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9504012510 con 0x7f95140f51a0 2024-01-18T06:52:21.802 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:21.802+0000 7f95197d8700 1 -- 172.21.15.194:0/2394112729 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9514107ef0 con 0x7f95140f51a0 2024-01-18T06:52:21.803 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:21.802+0000 7f9510ff9700 1 -- 172.21.15.194:0/2394112729 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9504012330 con 0x7f95140f51a0 2024-01-18T06:52:21.803 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:52:21.803 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:52:21.804 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:21.804+0000 7f95197d8700 1 -- 172.21.15.194:0/2394112729 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f94fc02ffb0 msgr2=0x7f94fc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:21.805 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:21.804+0000 7f95197d8700 1 --2- 172.21.15.194:0/2394112729 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f94fc02ffb0 0x7f94fc032470 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f95080096a0 tx=0x7f9508006b40).stop 2024-01-18T06:52:21.805 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:21.804+0000 7f95197d8700 1 -- 172.21.15.194:0/2394112729 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f95140f51a0 msgr2=0x7f95141064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:21.805 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:21.804+0000 7f95197d8700 1 --2- 172.21.15.194:0/2394112729 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f95140f51a0 0x7f95141064c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f9504013510 tx=0x7f950400b040).stop 2024-01-18T06:52:21.805 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:21.804+0000 7f95197d8700 1 -- 172.21.15.194:0/2394112729 shutdown_connections 2024-01-18T06:52:21.805 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:21.804+0000 7f95197d8700 1 --2- 172.21.15.194:0/2394112729 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f94fc02ffb0 0x7f94fc032470 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:21.805 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:21.804+0000 7f95197d8700 1 --2- 172.21.15.194:0/2394112729 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f95140f51a0 0x7f95141064c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:21.805 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:21.804+0000 7f95197d8700 1 -- 172.21.15.194:0/2394112729 >> 172.21.15.194:0/2394112729 conn(0x7f95140f09a0 msgr2=0x7f95140f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:52:21.805 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:21.804+0000 7f95197d8700 1 -- 172.21.15.194:0/2394112729 shutdown_connections 2024-01-18T06:52:21.805 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:21.804+0000 7f95197d8700 1 -- 172.21.15.194:0/2394112729 wait complete. 2024-01-18T06:52:21.805 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:52:22.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:21 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:52:21.804235+0000 mon.smithi047 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.194:0/2394112729' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:52:23.188 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:52:23.188 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:52:23.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:22 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:52:22.010011+0000 mgr.smithi047.fuhzzr (mgr.14162) 117 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:52:24.386 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:24.385+0000 7fbda69ad700 1 -- 172.21.15.194:0/52680013 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbda00f5410 msgr2=0x7fbda00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:24.386 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:24.385+0000 7fbda69ad700 1 --2- 172.21.15.194:0/52680013 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbda00f5410 0x7fbda00f5830 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7fbd88009fb0 tx=0x7fbd88009b30).stop 2024-01-18T06:52:24.387 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:24.385+0000 7fbda69ad700 1 -- 172.21.15.194:0/52680013 shutdown_connections 2024-01-18T06:52:24.387 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:24.385+0000 7fbda69ad700 1 --2- 172.21.15.194:0/52680013 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbda00f5410 0x7fbda00f5830 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:24.387 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:24.385+0000 7fbda69ad700 1 -- 172.21.15.194:0/52680013 >> 172.21.15.194:0/52680013 conn(0x7fbda00f0b70 msgr2=0x7fbda00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:52:24.387 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:24.385+0000 7fbda69ad700 1 -- 172.21.15.194:0/52680013 shutdown_connections 2024-01-18T06:52:24.387 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:24.385+0000 7fbda69ad700 1 -- 172.21.15.194:0/52680013 wait complete. 2024-01-18T06:52:24.387 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:24.386+0000 7fbda69ad700 1 Processor -- start 2024-01-18T06:52:24.387 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:24.386+0000 7fbda69ad700 1 -- start start 2024-01-18T06:52:24.387 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:24.386+0000 7fbda69ad700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbda00f5410 0x7fbda00fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:52:24.387 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:24.386+0000 7fbda69ad700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda00fb820 con 0x7fbda00f5410 2024-01-18T06:52:24.387 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:24.386+0000 7fbd9ffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbda00f5410 0x7fbda00fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:52:24.387 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:24.386+0000 7fbd9ffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbda00f5410 0x7fbda00fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:58938/0 (socket says 172.21.15.194:58938) 2024-01-18T06:52:24.387 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:24.386+0000 7fbd9ffff700 1 -- 172.21.15.194:0/801514665 learned_addr learned my addr 172.21.15.194:0/801514665 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:52:24.387 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:24.387+0000 7fbd9ffff700 1 -- 172.21.15.194:0/801514665 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd88005040 con 0x7fbda00f5410 2024-01-18T06:52:24.387 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:24.387+0000 7fbd9ffff700 1 --2- 172.21.15.194:0/801514665 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbda00f5410 0x7fbda00fd3a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fbd88013510 tx=0x7fbd8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:52:24.388 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:24.387+0000 7fbd9cff9700 1 -- 172.21.15.194:0/801514665 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd88009320 con 0x7fbda00f5410 2024-01-18T06:52:24.388 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:24.387+0000 7fbda69ad700 1 -- 172.21.15.194:0/801514665 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbda00fb9c0 con 0x7fbda00f5410 2024-01-18T06:52:24.388 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:24.387+0000 7fbd9cff9700 1 -- 172.21.15.194:0/801514665 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbd88013f00 con 0x7fbda00f5410 2024-01-18T06:52:24.388 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:24.387+0000 7fbd9cff9700 1 -- 172.21.15.194:0/801514665 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd88014d90 con 0x7fbda00f5410 2024-01-18T06:52:24.388 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:24.387+0000 7fbda69ad700 1 -- 172.21.15.194:0/801514665 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbda00fbe50 con 0x7fbda00f5410 2024-01-18T06:52:24.389 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:24.388+0000 7fbd9cff9700 1 -- 172.21.15.194:0/801514665 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbd880263a0 con 0x7fbda00f5410 2024-01-18T06:52:24.389 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:24.388+0000 7fbda69ad700 1 -- 172.21.15.194:0/801514665 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbda0059100 con 0x7fbda00f5410 2024-01-18T06:52:24.389 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:24.388+0000 7fbd9cff9700 1 --2- 172.21.15.194:0/801514665 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fbd8c02ff60 0x7fbd8c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:52:24.389 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:24.388+0000 7fbd9cff9700 1 -- 172.21.15.194:0/801514665 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbd88041380 con 0x7fbda00f5410 2024-01-18T06:52:24.389 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:24.388+0000 7fbd9f7fe700 1 --2- 172.21.15.194:0/801514665 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fbd8c02ff60 0x7fbd8c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:52:24.389 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:24.388+0000 7fbd9f7fe700 1 --2- 172.21.15.194:0/801514665 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fbd8c02ff60 0x7fbd8c032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fbd900096a0 tx=0x7fbd90006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:52:24.392 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:24.392+0000 7fbd9cff9700 1 -- 172.21.15.194:0/801514665 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbd8801cad0 con 0x7fbda00f5410 2024-01-18T06:52:24.596 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:24.597+0000 7fbda69ad700 1 -- 172.21.15.194:0/801514665 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbda00fae60 con 0x7fbda00f5410 2024-01-18T06:52:24.597 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:24.597+0000 7fbd9cff9700 1 -- 172.21.15.194:0/801514665 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbd8801cad0 con 0x7fbda00f5410 2024-01-18T06:52:24.597 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:52:24.597 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:52:24.598 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:24.599+0000 7fbda69ad700 1 -- 172.21.15.194:0/801514665 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fbd8c02ff60 msgr2=0x7fbd8c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:24.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:24.599+0000 7fbda69ad700 1 --2- 172.21.15.194:0/801514665 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fbd8c02ff60 0x7fbd8c032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fbd900096a0 tx=0x7fbd90006b40).stop 2024-01-18T06:52:24.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:24.599+0000 7fbda69ad700 1 -- 172.21.15.194:0/801514665 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbda00f5410 msgr2=0x7fbda00fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:24.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:24.599+0000 7fbda69ad700 1 --2- 172.21.15.194:0/801514665 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbda00f5410 0x7fbda00fd3a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fbd88013510 tx=0x7fbd8800b040).stop 2024-01-18T06:52:24.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:24.599+0000 7fbda69ad700 1 -- 172.21.15.194:0/801514665 shutdown_connections 2024-01-18T06:52:24.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:24.599+0000 7fbda69ad700 1 --2- 172.21.15.194:0/801514665 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fbd8c02ff60 0x7fbd8c032420 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:24.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:24.599+0000 7fbda69ad700 1 --2- 172.21.15.194:0/801514665 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbda00f5410 0x7fbda00fd3a0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:24.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:24.599+0000 7fbda69ad700 1 -- 172.21.15.194:0/801514665 >> 172.21.15.194:0/801514665 conn(0x7fbda00f0b70 msgr2=0x7fbda005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:52:24.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:24.599+0000 7fbda69ad700 1 -- 172.21.15.194:0/801514665 shutdown_connections 2024-01-18T06:52:24.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:24.599+0000 7fbda69ad700 1 -- 172.21.15.194:0/801514665 wait complete. 2024-01-18T06:52:24.599 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:52:25.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:25 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:52:24.010251+0000 mgr.smithi047.fuhzzr (mgr.14162) 118 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:52:25.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:25 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:52:24.598348+0000 mon.smithi047 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.194:0/801514665' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:52:26.022 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:52:26.022 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:52:27.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:27.202+0000 7f2d5fd5d700 1 -- 172.21.15.194:0/3085204974 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d580f6510 msgr2=0x7f2d580f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:27.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:27.202+0000 7f2d5fd5d700 1 --2- 172.21.15.194:0/3085204974 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d580f6510 0x7f2d580f6930 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f2d48009fb0 tx=0x7f2d48009b30).stop 2024-01-18T06:52:27.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:27.202+0000 7f2d5fd5d700 1 -- 172.21.15.194:0/3085204974 shutdown_connections 2024-01-18T06:52:27.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:27.202+0000 7f2d5fd5d700 1 --2- 172.21.15.194:0/3085204974 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d580f6510 0x7f2d580f6930 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:27.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:27.202+0000 7f2d5fd5d700 1 -- 172.21.15.194:0/3085204974 >> 172.21.15.194:0/3085204974 conn(0x7f2d580f1c70 msgr2=0x7f2d580f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:52:27.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:27.202+0000 7f2d5fd5d700 1 -- 172.21.15.194:0/3085204974 shutdown_connections 2024-01-18T06:52:27.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:27.202+0000 7f2d5fd5d700 1 -- 172.21.15.194:0/3085204974 wait complete. 2024-01-18T06:52:27.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:27.203+0000 7f2d5fd5d700 1 Processor -- start 2024-01-18T06:52:27.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:27.203+0000 7f2d5fd5d700 1 -- start start 2024-01-18T06:52:27.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:27.203+0000 7f2d5fd5d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d580f6510 0x7f2d58106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:52:27.205 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:27.203+0000 7f2d5fd5d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d58106c20 con 0x7f2d580f6510 2024-01-18T06:52:27.205 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:27.203+0000 7f2d5daf9700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d580f6510 0x7f2d58106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:52:27.205 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:27.203+0000 7f2d5daf9700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d580f6510 0x7f2d58106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:58948/0 (socket says 172.21.15.194:58948) 2024-01-18T06:52:27.205 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:27.203+0000 7f2d5daf9700 1 -- 172.21.15.194:0/322494382 learned_addr learned my addr 172.21.15.194:0/322494382 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:52:27.205 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:27.204+0000 7f2d5daf9700 1 -- 172.21.15.194:0/322494382 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d48005040 con 0x7f2d580f6510 2024-01-18T06:52:27.205 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:27.204+0000 7f2d5daf9700 1 --2- 172.21.15.194:0/322494382 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d580f6510 0x7f2d58106730 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f2d48013510 tx=0x7f2d4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:52:27.205 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:27.204+0000 7f2d4e7fc700 1 -- 172.21.15.194:0/322494382 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d48009320 con 0x7f2d580f6510 2024-01-18T06:52:27.205 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:27.204+0000 7f2d5fd5d700 1 -- 172.21.15.194:0/322494382 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d58106e20 con 0x7f2d580f6510 2024-01-18T06:52:27.205 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:27.204+0000 7f2d4e7fc700 1 -- 172.21.15.194:0/322494382 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2d48013f00 con 0x7f2d580f6510 2024-01-18T06:52:27.205 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:27.204+0000 7f2d5fd5d700 1 -- 172.21.15.194:0/322494382 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d581072b0 con 0x7f2d580f6510 2024-01-18T06:52:27.205 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:27.204+0000 7f2d4e7fc700 1 -- 172.21.15.194:0/322494382 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d48014d90 con 0x7f2d580f6510 2024-01-18T06:52:27.206 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:27.205+0000 7f2d4e7fc700 1 -- 172.21.15.194:0/322494382 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2d480095f0 con 0x7f2d580f6510 2024-01-18T06:52:27.206 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:27.205+0000 7f2d5fd5d700 1 -- 172.21.15.194:0/322494382 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d58045ad0 con 0x7f2d580f6510 2024-01-18T06:52:27.206 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:27.205+0000 7f2d4e7fc700 1 --2- 172.21.15.194:0/322494382 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f2d4402ff60 0x7f2d44032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:52:27.206 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:27.205+0000 7f2d4e7fc700 1 -- 172.21.15.194:0/322494382 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2d48040fd0 con 0x7f2d580f6510 2024-01-18T06:52:27.206 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:27.205+0000 7f2d5d2f8700 1 --2- 172.21.15.194:0/322494382 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f2d4402ff60 0x7f2d44032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:52:27.206 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:27.206+0000 7f2d5d2f8700 1 --2- 172.21.15.194:0/322494382 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f2d4402ff60 0x7f2d44032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f2d540096a0 tx=0x7f2d54006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:52:27.209 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:27.210+0000 7f2d4e7fc700 1 -- 172.21.15.194:0/322494382 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2d48014420 con 0x7f2d580f6510 2024-01-18T06:52:27.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:27.417+0000 7f2d5fd5d700 1 -- 172.21.15.194:0/322494382 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2d580fbf60 con 0x7f2d580f6510 2024-01-18T06:52:27.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:27.418+0000 7f2d4e7fc700 1 -- 172.21.15.194:0/322494382 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2d48010230 con 0x7f2d580f6510 2024-01-18T06:52:27.417 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:52:27.418 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:52:27.419 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:27.420+0000 7f2d5fd5d700 1 -- 172.21.15.194:0/322494382 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f2d4402ff60 msgr2=0x7f2d44032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:27.419 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:27.420+0000 7f2d5fd5d700 1 --2- 172.21.15.194:0/322494382 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f2d4402ff60 0x7f2d44032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f2d540096a0 tx=0x7f2d54006b40).stop 2024-01-18T06:52:27.419 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:27.420+0000 7f2d5fd5d700 1 -- 172.21.15.194:0/322494382 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d580f6510 msgr2=0x7f2d58106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:27.419 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:27.420+0000 7f2d5fd5d700 1 --2- 172.21.15.194:0/322494382 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d580f6510 0x7f2d58106730 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f2d48013510 tx=0x7f2d4800b040).stop 2024-01-18T06:52:27.419 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:27.420+0000 7f2d5fd5d700 1 -- 172.21.15.194:0/322494382 shutdown_connections 2024-01-18T06:52:27.420 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:27.420+0000 7f2d5fd5d700 1 --2- 172.21.15.194:0/322494382 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f2d4402ff60 0x7f2d44032420 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:27.420 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:27.420+0000 7f2d5fd5d700 1 --2- 172.21.15.194:0/322494382 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d580f6510 0x7f2d58106730 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:27.420 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:27.420+0000 7f2d5fd5d700 1 -- 172.21.15.194:0/322494382 >> 172.21.15.194:0/322494382 conn(0x7f2d580f1c70 msgr2=0x7f2d580f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:52:27.420 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:27.420+0000 7f2d5fd5d700 1 -- 172.21.15.194:0/322494382 shutdown_connections 2024-01-18T06:52:27.420 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:27.420+0000 7f2d5fd5d700 1 -- 172.21.15.194:0/322494382 wait complete. 2024-01-18T06:52:27.420 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:52:27.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:27 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:52:26.010623+0000 mgr.smithi047.fuhzzr (mgr.14162) 119 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:52:28.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:28 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:52:27.419006+0000 mon.smithi047 (mon.0) 196 : audit [DBG] from='client.? 172.21.15.194:0/322494382' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:52:28.802 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:52:28.802 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:52:29.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:29 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:52:28.010957+0000 mgr.smithi047.fuhzzr (mgr.14162) 120 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:52:30.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:30.025+0000 7fd21bbc6700 1 -- 172.21.15.194:0/1238203726 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2140f6510 msgr2=0x7fd2140f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:30.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:30.026+0000 7fd21bbc6700 1 --2- 172.21.15.194:0/1238203726 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2140f6510 0x7fd2140f6930 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7fd204009fb0 tx=0x7fd204009b30).stop 2024-01-18T06:52:30.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:30.026+0000 7fd21bbc6700 1 -- 172.21.15.194:0/1238203726 shutdown_connections 2024-01-18T06:52:30.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:30.026+0000 7fd21bbc6700 1 --2- 172.21.15.194:0/1238203726 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2140f6510 0x7fd2140f6930 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:30.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:30.026+0000 7fd21bbc6700 1 -- 172.21.15.194:0/1238203726 >> 172.21.15.194:0/1238203726 conn(0x7fd2140f1c70 msgr2=0x7fd2140f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:52:30.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:30.026+0000 7fd21bbc6700 1 -- 172.21.15.194:0/1238203726 shutdown_connections 2024-01-18T06:52:30.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:30.026+0000 7fd21bbc6700 1 -- 172.21.15.194:0/1238203726 wait complete. 2024-01-18T06:52:30.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:30.026+0000 7fd21bbc6700 1 Processor -- start 2024-01-18T06:52:30.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:30.026+0000 7fd21bbc6700 1 -- start start 2024-01-18T06:52:30.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:30.026+0000 7fd21bbc6700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2140f6510 0x7fd214106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:52:30.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:30.026+0000 7fd21bbc6700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd214106c20 con 0x7fd2140f6510 2024-01-18T06:52:30.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:30.026+0000 7fd219962700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2140f6510 0x7fd214106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:52:30.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:30.027+0000 7fd219962700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2140f6510 0x7fd214106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:38854/0 (socket says 172.21.15.194:38854) 2024-01-18T06:52:30.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:30.027+0000 7fd219962700 1 -- 172.21.15.194:0/3900066525 learned_addr learned my addr 172.21.15.194:0/3900066525 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:52:30.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:30.027+0000 7fd219962700 1 -- 172.21.15.194:0/3900066525 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd204005040 con 0x7fd2140f6510 2024-01-18T06:52:30.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:30.027+0000 7fd219962700 1 --2- 172.21.15.194:0/3900066525 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2140f6510 0x7fd214106730 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7fd204013510 tx=0x7fd20400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:52:30.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:30.027+0000 7fd20a7fc700 1 -- 172.21.15.194:0/3900066525 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd204009320 con 0x7fd2140f6510 2024-01-18T06:52:30.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:30.027+0000 7fd21bbc6700 1 -- 172.21.15.194:0/3900066525 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd214106e20 con 0x7fd2140f6510 2024-01-18T06:52:30.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:30.027+0000 7fd20a7fc700 1 -- 172.21.15.194:0/3900066525 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd204013f00 con 0x7fd2140f6510 2024-01-18T06:52:30.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:30.027+0000 7fd20a7fc700 1 -- 172.21.15.194:0/3900066525 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd204014d90 con 0x7fd2140f6510 2024-01-18T06:52:30.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:30.027+0000 7fd21bbc6700 1 -- 172.21.15.194:0/3900066525 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2141072b0 con 0x7fd2140f6510 2024-01-18T06:52:30.029 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:30.028+0000 7fd20a7fc700 1 -- 172.21.15.194:0/3900066525 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd2040095f0 con 0x7fd2140f6510 2024-01-18T06:52:30.029 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:30.028+0000 7fd21bbc6700 1 -- 172.21.15.194:0/3900066525 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd214045ad0 con 0x7fd2140f6510 2024-01-18T06:52:30.029 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:30.028+0000 7fd20a7fc700 1 --2- 172.21.15.194:0/3900066525 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fd20002ffb0 0x7fd200032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:52:30.029 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:30.028+0000 7fd20a7fc700 1 -- 172.21.15.194:0/3900066525 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd2040410a0 con 0x7fd2140f6510 2024-01-18T06:52:30.030 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:30.028+0000 7fd219161700 1 --2- 172.21.15.194:0/3900066525 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fd20002ffb0 0x7fd200032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:52:30.030 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:30.029+0000 7fd219161700 1 --2- 172.21.15.194:0/3900066525 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fd20002ffb0 0x7fd200032470 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fd2100096a0 tx=0x7fd210006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:52:30.032 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:30.033+0000 7fd20a7fc700 1 -- 172.21.15.194:0/3900066525 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd20406d2b0 con 0x7fd2140f6510 2024-01-18T06:52:30.235 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:30.235+0000 7fd21bbc6700 1 -- 172.21.15.194:0/3900066525 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd2140fbf60 con 0x7fd2140f6510 2024-01-18T06:52:30.236 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:30.236+0000 7fd20a7fc700 1 -- 172.21.15.194:0/3900066525 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd204014420 con 0x7fd2140f6510 2024-01-18T06:52:30.236 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:52:30.236 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:52:30.237 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:30.238+0000 7fd21bbc6700 1 -- 172.21.15.194:0/3900066525 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fd20002ffb0 msgr2=0x7fd200032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:30.238 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:30.238+0000 7fd21bbc6700 1 --2- 172.21.15.194:0/3900066525 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fd20002ffb0 0x7fd200032470 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fd2100096a0 tx=0x7fd210006b40).stop 2024-01-18T06:52:30.238 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:30.238+0000 7fd21bbc6700 1 -- 172.21.15.194:0/3900066525 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2140f6510 msgr2=0x7fd214106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:30.238 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:30.238+0000 7fd21bbc6700 1 --2- 172.21.15.194:0/3900066525 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2140f6510 0x7fd214106730 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7fd204013510 tx=0x7fd20400b040).stop 2024-01-18T06:52:30.238 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:30.238+0000 7fd21bbc6700 1 -- 172.21.15.194:0/3900066525 shutdown_connections 2024-01-18T06:52:30.238 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:30.238+0000 7fd21bbc6700 1 --2- 172.21.15.194:0/3900066525 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fd20002ffb0 0x7fd200032470 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:30.238 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:30.238+0000 7fd21bbc6700 1 --2- 172.21.15.194:0/3900066525 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2140f6510 0x7fd214106730 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:30.238 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:30.238+0000 7fd21bbc6700 1 -- 172.21.15.194:0/3900066525 >> 172.21.15.194:0/3900066525 conn(0x7fd2140f1c70 msgr2=0x7fd2140f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:52:30.238 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:30.238+0000 7fd21bbc6700 1 -- 172.21.15.194:0/3900066525 shutdown_connections 2024-01-18T06:52:30.238 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:30.238+0000 7fd21bbc6700 1 -- 172.21.15.194:0/3900066525 wait complete. 2024-01-18T06:52:30.238 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:52:31.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:31 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:52:30.011243+0000 mgr.smithi047.fuhzzr (mgr.14162) 121 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:52:31.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:31 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:52:30.114745+0000 mon.smithi047 (mon.0) 197 : audit [INF] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.fuhzzr/mirror_snapshot_schedule"}]: dispatch 2024-01-18T06:52:31.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:31 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:52:30.116255+0000 mon.smithi047 (mon.0) 198 : audit [INF] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.fuhzzr/trash_purge_schedule"}]: dispatch 2024-01-18T06:52:31.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:31 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:52:30.237110+0000 mon.smithi047 (mon.0) 199 : audit [DBG] from='client.? 172.21.15.194:0/3900066525' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:52:31.614 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:52:31.614 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:52:32.797 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:32.796+0000 7f9143fff700 1 -- 172.21.15.194:0/966478882 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91440601e0 msgr2=0x7f91440f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:32.797 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:32.796+0000 7f9143fff700 1 --2- 172.21.15.194:0/966478882 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91440601e0 0x7f91440f8f20 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f9138009fb0 tx=0x7f9138009b30).stop 2024-01-18T06:52:32.797 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:32.796+0000 7f9143fff700 1 -- 172.21.15.194:0/966478882 shutdown_connections 2024-01-18T06:52:32.797 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:32.796+0000 7f9143fff700 1 --2- 172.21.15.194:0/966478882 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91440601e0 0x7f91440f8f20 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:32.797 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:32.796+0000 7f9143fff700 1 -- 172.21.15.194:0/966478882 >> 172.21.15.194:0/966478882 conn(0x7f91440f09a0 msgr2=0x7f91440f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:52:32.798 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:32.796+0000 7f9143fff700 1 -- 172.21.15.194:0/966478882 shutdown_connections 2024-01-18T06:52:32.798 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:32.796+0000 7f9143fff700 1 -- 172.21.15.194:0/966478882 wait complete. 2024-01-18T06:52:32.798 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:32.796+0000 7f9143fff700 1 Processor -- start 2024-01-18T06:52:32.798 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:32.796+0000 7f9143fff700 1 -- start start 2024-01-18T06:52:32.798 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:32.797+0000 7f9143fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91440601e0 0x7f9144106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:52:32.798 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:32.797+0000 7f9143fff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91441069f0 con 0x7f91440601e0 2024-01-18T06:52:32.798 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:32.797+0000 7f9142ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91440601e0 0x7f9144106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:52:32.798 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:32.797+0000 7f9142ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91440601e0 0x7f9144106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:38868/0 (socket says 172.21.15.194:38868) 2024-01-18T06:52:32.798 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:32.797+0000 7f9142ffd700 1 -- 172.21.15.194:0/3113488343 learned_addr learned my addr 172.21.15.194:0/3113488343 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:52:32.798 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:32.797+0000 7f9142ffd700 1 -- 172.21.15.194:0/3113488343 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9138005040 con 0x7f91440601e0 2024-01-18T06:52:32.798 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:32.797+0000 7f9142ffd700 1 --2- 172.21.15.194:0/3113488343 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91440601e0 0x7f9144106500 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f9138013510 tx=0x7f913800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:52:32.798 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:32.797+0000 7f912b7fe700 1 -- 172.21.15.194:0/3113488343 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9138009320 con 0x7f91440601e0 2024-01-18T06:52:32.798 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:32.797+0000 7f9143fff700 1 -- 172.21.15.194:0/3113488343 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9144106bf0 con 0x7f91440601e0 2024-01-18T06:52:32.798 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:32.797+0000 7f912b7fe700 1 -- 172.21.15.194:0/3113488343 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9138013f00 con 0x7f91440601e0 2024-01-18T06:52:32.798 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:32.797+0000 7f912b7fe700 1 -- 172.21.15.194:0/3113488343 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9138014d90 con 0x7f91440601e0 2024-01-18T06:52:32.798 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:32.797+0000 7f9143fff700 1 -- 172.21.15.194:0/3113488343 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9144107080 con 0x7f91440601e0 2024-01-18T06:52:32.799 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:32.798+0000 7f912b7fe700 1 -- 172.21.15.194:0/3113488343 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9138014420 con 0x7f91440601e0 2024-01-18T06:52:32.799 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:32.798+0000 7f9143fff700 1 -- 172.21.15.194:0/3113488343 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9144045ad0 con 0x7f91440601e0 2024-01-18T06:52:32.800 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:32.798+0000 7f912b7fe700 1 --2- 172.21.15.194:0/3113488343 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f912c02ff60 0x7f912c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:52:32.800 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:32.798+0000 7f912b7fe700 1 -- 172.21.15.194:0/3113488343 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9138041070 con 0x7f91440601e0 2024-01-18T06:52:32.800 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:32.799+0000 7f91427fc700 1 --2- 172.21.15.194:0/3113488343 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f912c02ff60 0x7f912c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:52:32.800 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:32.799+0000 7f91427fc700 1 --2- 172.21.15.194:0/3113488343 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f912c02ff60 0x7f912c032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f91340096a0 tx=0x7f9134006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:52:32.802 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:32.803+0000 7f912b7fe700 1 -- 172.21.15.194:0/3113488343 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f91380095f0 con 0x7f91440601e0 2024-01-18T06:52:33.008 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:33.009+0000 7f9143fff700 1 -- 172.21.15.194:0/3113488343 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9144107f30 con 0x7f91440601e0 2024-01-18T06:52:33.009 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:33.010+0000 7f912b7fe700 1 -- 172.21.15.194:0/3113488343 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f913801db70 con 0x7f91440601e0 2024-01-18T06:52:33.009 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:52:33.009 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:52:33.011 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:33.011+0000 7f9143fff700 1 -- 172.21.15.194:0/3113488343 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f912c02ff60 msgr2=0x7f912c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:33.011 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:33.011+0000 7f9143fff700 1 --2- 172.21.15.194:0/3113488343 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f912c02ff60 0x7f912c032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f91340096a0 tx=0x7f9134006b40).stop 2024-01-18T06:52:33.011 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:33.011+0000 7f9143fff700 1 -- 172.21.15.194:0/3113488343 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91440601e0 msgr2=0x7f9144106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:33.011 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:33.011+0000 7f9143fff700 1 --2- 172.21.15.194:0/3113488343 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91440601e0 0x7f9144106500 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f9138013510 tx=0x7f913800b040).stop 2024-01-18T06:52:33.011 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:33.011+0000 7f9143fff700 1 -- 172.21.15.194:0/3113488343 shutdown_connections 2024-01-18T06:52:33.011 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:33.011+0000 7f9143fff700 1 --2- 172.21.15.194:0/3113488343 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f912c02ff60 0x7f912c032420 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:33.011 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:33.011+0000 7f9143fff700 1 --2- 172.21.15.194:0/3113488343 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91440601e0 0x7f9144106500 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:33.011 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:33.011+0000 7f9143fff700 1 -- 172.21.15.194:0/3113488343 >> 172.21.15.194:0/3113488343 conn(0x7f91440f09a0 msgr2=0x7f91440f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:52:33.011 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:33.011+0000 7f9143fff700 1 -- 172.21.15.194:0/3113488343 shutdown_connections 2024-01-18T06:52:33.011 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:33.011+0000 7f9143fff700 1 -- 172.21.15.194:0/3113488343 wait complete. 2024-01-18T06:52:33.012 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:52:33.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:33 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:52:32.011604+0000 mgr.smithi047.fuhzzr (mgr.14162) 122 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:52:33.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:33 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:52:33.010421+0000 mon.smithi047 (mon.0) 200 : audit [DBG] from='client.? 172.21.15.194:0/3113488343' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:52:34.404 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:52:34.405 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:52:35.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:35 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:52:34.011933+0000 mgr.smithi047.fuhzzr (mgr.14162) 123 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:52:35.568 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:35.569+0000 7f0b69ded700 1 -- 172.21.15.194:0/32250230 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b64060280 msgr2=0x7f0b640f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:35.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:35.569+0000 7f0b69ded700 1 --2- 172.21.15.194:0/32250230 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b64060280 0x7f0b640f90f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f0b4c009fb0 tx=0x7f0b4c009b30).stop 2024-01-18T06:52:35.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:35.569+0000 7f0b69ded700 1 -- 172.21.15.194:0/32250230 shutdown_connections 2024-01-18T06:52:35.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:35.569+0000 7f0b69ded700 1 --2- 172.21.15.194:0/32250230 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b64060280 0x7f0b640f90f0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:35.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:35.569+0000 7f0b69ded700 1 -- 172.21.15.194:0/32250230 >> 172.21.15.194:0/32250230 conn(0x7f0b640f0b70 msgr2=0x7f0b640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:52:35.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:35.569+0000 7f0b69ded700 1 -- 172.21.15.194:0/32250230 shutdown_connections 2024-01-18T06:52:35.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:35.569+0000 7f0b69ded700 1 -- 172.21.15.194:0/32250230 wait complete. 2024-01-18T06:52:35.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:35.569+0000 7f0b69ded700 1 Processor -- start 2024-01-18T06:52:35.570 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:35.570+0000 7f0b69ded700 1 -- start start 2024-01-18T06:52:35.570 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:35.570+0000 7f0b69ded700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b64060280 0x7f0b640f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:52:35.570 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:35.570+0000 7f0b69ded700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b640f4ff0 con 0x7f0b64060280 2024-01-18T06:52:35.571 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:35.570+0000 7f0b637fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b64060280 0x7f0b640f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:52:35.571 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:35.570+0000 7f0b637fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b64060280 0x7f0b640f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:38886/0 (socket says 172.21.15.194:38886) 2024-01-18T06:52:35.571 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:35.570+0000 7f0b637fe700 1 -- 172.21.15.194:0/1602889619 learned_addr learned my addr 172.21.15.194:0/1602889619 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:52:35.571 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:35.570+0000 7f0b637fe700 1 -- 172.21.15.194:0/1602889619 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b4c005040 con 0x7f0b64060280 2024-01-18T06:52:35.571 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:35.570+0000 7f0b637fe700 1 --2- 172.21.15.194:0/1602889619 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b64060280 0x7f0b640f6b70 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f0b4c013510 tx=0x7f0b4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:52:35.571 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:35.571+0000 7f0b5bfff700 1 -- 172.21.15.194:0/1602889619 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b4c009320 con 0x7f0b64060280 2024-01-18T06:52:35.571 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:35.571+0000 7f0b69ded700 1 -- 172.21.15.194:0/1602889619 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b640f5190 con 0x7f0b64060280 2024-01-18T06:52:35.571 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:35.571+0000 7f0b5bfff700 1 -- 172.21.15.194:0/1602889619 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b4c013f00 con 0x7f0b64060280 2024-01-18T06:52:35.571 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:35.571+0000 7f0b5bfff700 1 -- 172.21.15.194:0/1602889619 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b4c014dc0 con 0x7f0b64060280 2024-01-18T06:52:35.571 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:35.571+0000 7f0b69ded700 1 -- 172.21.15.194:0/1602889619 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b640f5620 con 0x7f0b64060280 2024-01-18T06:52:35.572 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:35.572+0000 7f0b5bfff700 1 -- 172.21.15.194:0/1602889619 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0b4c0095f0 con 0x7f0b64060280 2024-01-18T06:52:35.572 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:35.572+0000 7f0b69ded700 1 -- 172.21.15.194:0/1602889619 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b64059100 con 0x7f0b64060280 2024-01-18T06:52:35.572 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:35.572+0000 7f0b5bfff700 1 --2- 172.21.15.194:0/1602889619 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f0b5002ff60 0x7f0b50032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:52:35.572 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:35.572+0000 7f0b5bfff700 1 -- 172.21.15.194:0/1602889619 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0b4c0410c0 con 0x7f0b64060280 2024-01-18T06:52:35.572 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:35.572+0000 7f0b62ffd700 1 --2- 172.21.15.194:0/1602889619 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f0b5002ff60 0x7f0b50032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:52:35.572 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:35.572+0000 7f0b62ffd700 1 --2- 172.21.15.194:0/1602889619 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f0b5002ff60 0x7f0b50032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f0b540096a0 tx=0x7f0b54006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:52:35.576 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:35.576+0000 7f0b5bfff700 1 -- 172.21.15.194:0/1602889619 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0b4c01c790 con 0x7f0b64060280 2024-01-18T06:52:35.784 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:35.785+0000 7f0b69ded700 1 -- 172.21.15.194:0/1602889619 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0b640fbf50 con 0x7f0b64060280 2024-01-18T06:52:35.785 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:35.785+0000 7f0b5bfff700 1 -- 172.21.15.194:0/1602889619 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0b4c010230 con 0x7f0b64060280 2024-01-18T06:52:35.785 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:52:35.785 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:52:35.786 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:35.787+0000 7f0b69ded700 1 -- 172.21.15.194:0/1602889619 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f0b5002ff60 msgr2=0x7f0b50032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:35.786 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:35.787+0000 7f0b69ded700 1 --2- 172.21.15.194:0/1602889619 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f0b5002ff60 0x7f0b50032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f0b540096a0 tx=0x7f0b54006b40).stop 2024-01-18T06:52:35.786 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:35.787+0000 7f0b69ded700 1 -- 172.21.15.194:0/1602889619 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b64060280 msgr2=0x7f0b640f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:35.787 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:35.787+0000 7f0b69ded700 1 --2- 172.21.15.194:0/1602889619 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b64060280 0x7f0b640f6b70 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f0b4c013510 tx=0x7f0b4c00b040).stop 2024-01-18T06:52:35.787 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:35.787+0000 7f0b69ded700 1 -- 172.21.15.194:0/1602889619 shutdown_connections 2024-01-18T06:52:35.787 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:35.787+0000 7f0b69ded700 1 --2- 172.21.15.194:0/1602889619 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f0b5002ff60 0x7f0b50032420 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:35.787 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:35.787+0000 7f0b69ded700 1 --2- 172.21.15.194:0/1602889619 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b64060280 0x7f0b640f6b70 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:35.787 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:35.787+0000 7f0b69ded700 1 -- 172.21.15.194:0/1602889619 >> 172.21.15.194:0/1602889619 conn(0x7f0b640f0b70 msgr2=0x7f0b640f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:52:35.787 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:35.787+0000 7f0b69ded700 1 -- 172.21.15.194:0/1602889619 shutdown_connections 2024-01-18T06:52:35.787 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:35.787+0000 7f0b69ded700 1 -- 172.21.15.194:0/1602889619 wait complete. 2024-01-18T06:52:35.787 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:52:36.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:36 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:52:35.786163+0000 mon.smithi047 (mon.0) 201 : audit [DBG] from='client.? 172.21.15.194:0/1602889619' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:52:37.167 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:52:37.167 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:52:37.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:52:36.012189+0000 mgr.smithi047.fuhzzr (mgr.14162) 124 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:52:38.322 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:38.322+0000 7fb131e3c700 1 -- 172.21.15.194:0/1536764120 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb12c0f6510 msgr2=0x7fb12c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:38.322 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:38.322+0000 7fb131e3c700 1 --2- 172.21.15.194:0/1536764120 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb12c0f6510 0x7fb12c0f6930 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7fb11c009fb0 tx=0x7fb11c009b30).stop 2024-01-18T06:52:38.322 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:38.322+0000 7fb131e3c700 1 -- 172.21.15.194:0/1536764120 shutdown_connections 2024-01-18T06:52:38.322 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:38.322+0000 7fb131e3c700 1 --2- 172.21.15.194:0/1536764120 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb12c0f6510 0x7fb12c0f6930 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:38.322 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:38.322+0000 7fb131e3c700 1 -- 172.21.15.194:0/1536764120 >> 172.21.15.194:0/1536764120 conn(0x7fb12c0f1c70 msgr2=0x7fb12c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:52:38.322 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:38.322+0000 7fb131e3c700 1 -- 172.21.15.194:0/1536764120 shutdown_connections 2024-01-18T06:52:38.323 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:38.323+0000 7fb131e3c700 1 -- 172.21.15.194:0/1536764120 wait complete. 2024-01-18T06:52:38.324 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:38.323+0000 7fb131e3c700 1 Processor -- start 2024-01-18T06:52:38.324 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:38.323+0000 7fb131e3c700 1 -- start start 2024-01-18T06:52:38.324 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:38.324+0000 7fb131e3c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb12c0f6510 0x7fb12c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:52:38.324 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:38.324+0000 7fb131e3c700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb12c106c20 con 0x7fb12c0f6510 2024-01-18T06:52:38.324 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:38.324+0000 7fb12b7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb12c0f6510 0x7fb12c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:52:38.325 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:38.324+0000 7fb12b7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb12c0f6510 0x7fb12c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:44638/0 (socket says 172.21.15.194:44638) 2024-01-18T06:52:38.325 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:38.324+0000 7fb12b7fe700 1 -- 172.21.15.194:0/1945842877 learned_addr learned my addr 172.21.15.194:0/1945842877 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:52:38.325 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:38.324+0000 7fb12b7fe700 1 -- 172.21.15.194:0/1945842877 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb11c005040 con 0x7fb12c0f6510 2024-01-18T06:52:38.325 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:38.325+0000 7fb12b7fe700 1 --2- 172.21.15.194:0/1945842877 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb12c0f6510 0x7fb12c106730 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fb11c014000 tx=0x7fb11c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:52:38.325 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:38.325+0000 7fb113fff700 1 -- 172.21.15.194:0/1945842877 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb11c0095f0 con 0x7fb12c0f6510 2024-01-18T06:52:38.325 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:38.325+0000 7fb131e3c700 1 -- 172.21.15.194:0/1945842877 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb12c106e20 con 0x7fb12c0f6510 2024-01-18T06:52:38.325 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:38.325+0000 7fb113fff700 1 -- 172.21.15.194:0/1945842877 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb11c007e70 con 0x7fb12c0f6510 2024-01-18T06:52:38.325 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:38.325+0000 7fb113fff700 1 -- 172.21.15.194:0/1945842877 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb11c014ea0 con 0x7fb12c0f6510 2024-01-18T06:52:38.325 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:38.325+0000 7fb131e3c700 1 -- 172.21.15.194:0/1945842877 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb12c1072b0 con 0x7fb12c0f6510 2024-01-18T06:52:38.326 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:38.326+0000 7fb113fff700 1 -- 172.21.15.194:0/1945842877 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb11c025070 con 0x7fb12c0f6510 2024-01-18T06:52:38.326 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:38.326+0000 7fb113fff700 1 --2- 172.21.15.194:0/1945842877 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb11402ff60 0x7fb114032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:52:38.326 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:38.326+0000 7fb113fff700 1 -- 172.21.15.194:0/1945842877 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb11c040980 con 0x7fb12c0f6510 2024-01-18T06:52:38.326 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:38.326+0000 7fb131e3c700 1 -- 172.21.15.194:0/1945842877 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb12c045ad0 con 0x7fb12c0f6510 2024-01-18T06:52:38.326 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:38.326+0000 7fb12affd700 1 --2- 172.21.15.194:0/1945842877 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb11402ff60 0x7fb114032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:52:38.326 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:38.326+0000 7fb12affd700 1 --2- 172.21.15.194:0/1945842877 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb11402ff60 0x7fb114032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fb1200096a0 tx=0x7fb120006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:52:38.330 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:38.331+0000 7fb113fff700 1 -- 172.21.15.194:0/1945842877 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb11c026350 con 0x7fb12c0f6510 2024-01-18T06:52:38.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:38.539+0000 7fb131e3c700 1 -- 172.21.15.194:0/1945842877 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb12c107fd0 con 0x7fb12c0f6510 2024-01-18T06:52:38.539 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:38.540+0000 7fb113fff700 1 -- 172.21.15.194:0/1945842877 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb11c012330 con 0x7fb12c0f6510 2024-01-18T06:52:38.539 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:52:38.539 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:52:38.541 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:38.541+0000 7fb131e3c700 1 -- 172.21.15.194:0/1945842877 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb11402ff60 msgr2=0x7fb114032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:38.541 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:38.541+0000 7fb131e3c700 1 --2- 172.21.15.194:0/1945842877 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb11402ff60 0x7fb114032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fb1200096a0 tx=0x7fb120006b40).stop 2024-01-18T06:52:38.541 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:38.541+0000 7fb131e3c700 1 -- 172.21.15.194:0/1945842877 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb12c0f6510 msgr2=0x7fb12c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:38.541 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:38.541+0000 7fb131e3c700 1 --2- 172.21.15.194:0/1945842877 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb12c0f6510 0x7fb12c106730 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fb11c014000 tx=0x7fb11c00b040).stop 2024-01-18T06:52:38.541 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:38.541+0000 7fb131e3c700 1 -- 172.21.15.194:0/1945842877 shutdown_connections 2024-01-18T06:52:38.541 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:38.541+0000 7fb131e3c700 1 --2- 172.21.15.194:0/1945842877 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb11402ff60 0x7fb114032420 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:38.541 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:38.541+0000 7fb131e3c700 1 --2- 172.21.15.194:0/1945842877 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb12c0f6510 0x7fb12c106730 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:38.541 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:38.541+0000 7fb131e3c700 1 -- 172.21.15.194:0/1945842877 >> 172.21.15.194:0/1945842877 conn(0x7fb12c0f1c70 msgr2=0x7fb12c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:52:38.541 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:38.541+0000 7fb131e3c700 1 -- 172.21.15.194:0/1945842877 shutdown_connections 2024-01-18T06:52:38.542 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:38.541+0000 7fb131e3c700 1 -- 172.21.15.194:0/1945842877 wait complete. 2024-01-18T06:52:38.542 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:52:39.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:38 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:52:38.012517+0000 mgr.smithi047.fuhzzr (mgr.14162) 125 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:52:39.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:38 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:52:38.540747+0000 mon.smithi047 (mon.0) 202 : audit [DBG] from='client.? 172.21.15.194:0/1945842877' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:52:39.943 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:52:39.944 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:52:41.141 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:41.141+0000 7f67a1f22700 1 -- 172.21.15.194:0/4212876963 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f679c0f62a0 msgr2=0x7f679c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:41.142 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:41.141+0000 7f67a1f22700 1 --2- 172.21.15.194:0/4212876963 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f679c0f62a0 0x7f679c0f66c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f6790009fb0 tx=0x7f6790009b30).stop 2024-01-18T06:52:41.142 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:41.142+0000 7f67a1f22700 1 -- 172.21.15.194:0/4212876963 shutdown_connections 2024-01-18T06:52:41.142 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:41.142+0000 7f67a1f22700 1 --2- 172.21.15.194:0/4212876963 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f679c0f62a0 0x7f679c0f66c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:41.142 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:41.142+0000 7f67a1f22700 1 -- 172.21.15.194:0/4212876963 >> 172.21.15.194:0/4212876963 conn(0x7f679c0f1aa0 msgr2=0x7f679c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:52:41.142 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:41.142+0000 7f67a1f22700 1 -- 172.21.15.194:0/4212876963 shutdown_connections 2024-01-18T06:52:41.142 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:41.142+0000 7f67a1f22700 1 -- 172.21.15.194:0/4212876963 wait complete. 2024-01-18T06:52:41.142 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:41.142+0000 7f67a1f22700 1 Processor -- start 2024-01-18T06:52:41.142 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:41.142+0000 7f67a1f22700 1 -- start start 2024-01-18T06:52:41.144 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:41.142+0000 7f67a1f22700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f679c0f62a0 0x7f679c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:52:41.144 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:41.142+0000 7f67a1f22700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679c069e20 con 0x7f679c0f62a0 2024-01-18T06:52:41.144 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:41.143+0000 7f67a0f20700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f679c0f62a0 0x7f679c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:52:41.144 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:41.143+0000 7f67a0f20700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f679c0f62a0 0x7f679c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:44656/0 (socket says 172.21.15.194:44656) 2024-01-18T06:52:41.144 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:41.143+0000 7f67a0f20700 1 -- 172.21.15.194:0/4141173558 learned_addr learned my addr 172.21.15.194:0/4141173558 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:52:41.144 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:41.143+0000 7f67a0f20700 1 -- 172.21.15.194:0/4141173558 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6790005040 con 0x7f679c0f62a0 2024-01-18T06:52:41.144 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:41.143+0000 7f67a0f20700 1 --2- 172.21.15.194:0/4141173558 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f679c0f62a0 0x7f679c06b9a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f6790013510 tx=0x7f679000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:52:41.144 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:41.143+0000 7f67997fa700 1 -- 172.21.15.194:0/4141173558 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6790009320 con 0x7f679c0f62a0 2024-01-18T06:52:41.144 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:41.143+0000 7f67a1f22700 1 -- 172.21.15.194:0/4141173558 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f679c069fc0 con 0x7f679c0f62a0 2024-01-18T06:52:41.144 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:41.143+0000 7f67997fa700 1 -- 172.21.15.194:0/4141173558 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6790013f00 con 0x7f679c0f62a0 2024-01-18T06:52:41.144 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:41.143+0000 7f67997fa700 1 -- 172.21.15.194:0/4141173558 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6790014dc0 con 0x7f679c0f62a0 2024-01-18T06:52:41.144 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:41.143+0000 7f67a1f22700 1 -- 172.21.15.194:0/4141173558 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f679c06a450 con 0x7f679c0f62a0 2024-01-18T06:52:41.145 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:41.144+0000 7f67997fa700 1 -- 172.21.15.194:0/4141173558 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f679001d3d0 con 0x7f679c0f62a0 2024-01-18T06:52:41.145 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:41.144+0000 7f67997fa700 1 --2- 172.21.15.194:0/4141173558 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f678c02ff60 0x7f678c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:52:41.145 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:41.144+0000 7f67997fa700 1 -- 172.21.15.194:0/4141173558 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f67900410e0 con 0x7f679c0f62a0 2024-01-18T06:52:41.145 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:41.144+0000 7f67a1f22700 1 -- 172.21.15.194:0/4141173558 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f679c059100 con 0x7f679c0f62a0 2024-01-18T06:52:41.145 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:41.145+0000 7f679bfff700 1 --2- 172.21.15.194:0/4141173558 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f678c02ff60 0x7f678c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:52:41.145 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:41.145+0000 7f679bfff700 1 --2- 172.21.15.194:0/4141173558 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f678c02ff60 0x7f678c032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f67880096a0 tx=0x7f6788006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:52:41.148 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:41.149+0000 7f67997fa700 1 -- 172.21.15.194:0/4141173558 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6790012920 con 0x7f679c0f62a0 2024-01-18T06:52:41.353 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:41.353+0000 7f67a1f22700 1 -- 172.21.15.194:0/4141173558 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f679c06ab50 con 0x7f679c0f62a0 2024-01-18T06:52:41.353 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:41.354+0000 7f67997fa700 1 -- 172.21.15.194:0/4141173558 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f679002b020 con 0x7f679c0f62a0 2024-01-18T06:52:41.354 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:52:41.354 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:52:41.355 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:41.356+0000 7f67a1f22700 1 -- 172.21.15.194:0/4141173558 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f678c02ff60 msgr2=0x7f678c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:41.356 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:41.356+0000 7f67a1f22700 1 --2- 172.21.15.194:0/4141173558 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f678c02ff60 0x7f678c032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f67880096a0 tx=0x7f6788006b40).stop 2024-01-18T06:52:41.356 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:41.356+0000 7f67a1f22700 1 -- 172.21.15.194:0/4141173558 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f679c0f62a0 msgr2=0x7f679c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:41.356 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:41.356+0000 7f67a1f22700 1 --2- 172.21.15.194:0/4141173558 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f679c0f62a0 0x7f679c06b9a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f6790013510 tx=0x7f679000b040).stop 2024-01-18T06:52:41.356 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:41.356+0000 7f67a1f22700 1 -- 172.21.15.194:0/4141173558 shutdown_connections 2024-01-18T06:52:41.356 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:41.356+0000 7f67a1f22700 1 --2- 172.21.15.194:0/4141173558 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f678c02ff60 0x7f678c032420 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:41.356 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:41.356+0000 7f67a1f22700 1 --2- 172.21.15.194:0/4141173558 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f679c0f62a0 0x7f679c06b9a0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:41.356 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:41.356+0000 7f67a1f22700 1 -- 172.21.15.194:0/4141173558 >> 172.21.15.194:0/4141173558 conn(0x7f679c0f1aa0 msgr2=0x7f679c0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:52:41.356 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:41.356+0000 7f67a1f22700 1 -- 172.21.15.194:0/4141173558 shutdown_connections 2024-01-18T06:52:41.356 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:41.356+0000 7f67a1f22700 1 -- 172.21.15.194:0/4141173558 wait complete. 2024-01-18T06:52:41.357 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:52:41.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:41 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:52:40.012861+0000 mgr.smithi047.fuhzzr (mgr.14162) 126 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:52:42.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:42 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:52:41.355136+0000 mon.smithi047 (mon.0) 203 : audit [DBG] from='client.? 172.21.15.194:0/4141173558' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:52:42.737 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:52:42.738 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:52:43.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:43 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:52:42.013118+0000 mgr.smithi047.fuhzzr (mgr.14162) 127 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:52:43.903 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:43.904+0000 7f7ab1b50700 1 -- 172.21.15.194:0/36886845 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7aac0f5410 msgr2=0x7f7aac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:43.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:43.904+0000 7f7ab1b50700 1 --2- 172.21.15.194:0/36886845 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7aac0f5410 0x7f7aac0f5830 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f7a9c009fb0 tx=0x7f7a9c009b30).stop 2024-01-18T06:52:43.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:43.904+0000 7f7ab1b50700 1 -- 172.21.15.194:0/36886845 shutdown_connections 2024-01-18T06:52:43.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:43.904+0000 7f7ab1b50700 1 --2- 172.21.15.194:0/36886845 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7aac0f5410 0x7f7aac0f5830 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:43.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:43.904+0000 7f7ab1b50700 1 -- 172.21.15.194:0/36886845 >> 172.21.15.194:0/36886845 conn(0x7f7aac0f0b70 msgr2=0x7f7aac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:52:43.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:43.904+0000 7f7ab1b50700 1 -- 172.21.15.194:0/36886845 shutdown_connections 2024-01-18T06:52:43.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:43.904+0000 7f7ab1b50700 1 -- 172.21.15.194:0/36886845 wait complete. 2024-01-18T06:52:43.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:43.904+0000 7f7ab1b50700 1 Processor -- start 2024-01-18T06:52:43.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:43.904+0000 7f7ab1b50700 1 -- start start 2024-01-18T06:52:43.905 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:43.904+0000 7f7ab1b50700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7aac0f5410 0x7f7aac109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:52:43.905 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:43.904+0000 7f7ab1b50700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aac060220 con 0x7f7aac0f5410 2024-01-18T06:52:43.906 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:43.905+0000 7f7aab7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7aac0f5410 0x7f7aac109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:52:43.906 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:43.905+0000 7f7aab7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7aac0f5410 0x7f7aac109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:44680/0 (socket says 172.21.15.194:44680) 2024-01-18T06:52:43.906 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:43.905+0000 7f7aab7fe700 1 -- 172.21.15.194:0/4010106428 learned_addr learned my addr 172.21.15.194:0/4010106428 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:52:43.906 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:43.905+0000 7f7aab7fe700 1 -- 172.21.15.194:0/4010106428 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a9c005040 con 0x7f7aac0f5410 2024-01-18T06:52:43.906 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:43.905+0000 7f7aab7fe700 1 --2- 172.21.15.194:0/4010106428 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7aac0f5410 0x7f7aac109450 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f7a9c013510 tx=0x7f7a9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:52:43.906 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:43.905+0000 7f7a93fff700 1 -- 172.21.15.194:0/4010106428 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a9c009320 con 0x7f7aac0f5410 2024-01-18T06:52:43.906 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:43.905+0000 7f7ab1b50700 1 -- 172.21.15.194:0/4010106428 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aac060420 con 0x7f7aac0f5410 2024-01-18T06:52:43.906 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:43.905+0000 7f7a93fff700 1 -- 172.21.15.194:0/4010106428 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7a9c013f00 con 0x7f7aac0f5410 2024-01-18T06:52:43.906 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:43.906+0000 7f7a93fff700 1 -- 172.21.15.194:0/4010106428 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a9c014d90 con 0x7f7aac0f5410 2024-01-18T06:52:43.906 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:43.906+0000 7f7ab1b50700 1 -- 172.21.15.194:0/4010106428 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aac109b00 con 0x7f7aac0f5410 2024-01-18T06:52:43.906 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:43.906+0000 7f7a93fff700 1 -- 172.21.15.194:0/4010106428 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7a9c0263a0 con 0x7f7aac0f5410 2024-01-18T06:52:43.907 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:43.906+0000 7f7a93fff700 1 --2- 172.21.15.194:0/4010106428 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f7a9402ff60 0x7f7a94032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:52:43.907 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:43.907+0000 7f7a93fff700 1 -- 172.21.15.194:0/4010106428 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7a9c0410a0 con 0x7f7aac0f5410 2024-01-18T06:52:43.907 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:43.907+0000 7f7ab1b50700 1 -- 172.21.15.194:0/4010106428 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7aac059100 con 0x7f7aac0f5410 2024-01-18T06:52:43.907 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:43.907+0000 7f7aaaffd700 1 --2- 172.21.15.194:0/4010106428 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f7a9402ff60 0x7f7a94032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:52:43.907 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:43.907+0000 7f7aaaffd700 1 --2- 172.21.15.194:0/4010106428 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f7a9402ff60 0x7f7a94032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f7aa00096a0 tx=0x7f7aa0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:52:43.911 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:43.912+0000 7f7a93fff700 1 -- 172.21.15.194:0/4010106428 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7a9c012920 con 0x7f7aac0f5410 2024-01-18T06:52:44.120 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:44.121+0000 7f7ab1b50700 1 -- 172.21.15.194:0/4010106428 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7aac045ad0 con 0x7f7aac0f5410 2024-01-18T06:52:44.121 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:44.121+0000 7f7a93fff700 1 -- 172.21.15.194:0/4010106428 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7a9c00f340 con 0x7f7aac0f5410 2024-01-18T06:52:44.121 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:52:44.121 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:52:44.122 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:44.123+0000 7f7ab1b50700 1 -- 172.21.15.194:0/4010106428 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f7a9402ff60 msgr2=0x7f7a94032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:44.122 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:44.123+0000 7f7ab1b50700 1 --2- 172.21.15.194:0/4010106428 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f7a9402ff60 0x7f7a94032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f7aa00096a0 tx=0x7f7aa0006b40).stop 2024-01-18T06:52:44.123 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:44.123+0000 7f7ab1b50700 1 -- 172.21.15.194:0/4010106428 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7aac0f5410 msgr2=0x7f7aac109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:44.123 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:44.123+0000 7f7ab1b50700 1 --2- 172.21.15.194:0/4010106428 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7aac0f5410 0x7f7aac109450 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f7a9c013510 tx=0x7f7a9c00b040).stop 2024-01-18T06:52:44.123 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:44.123+0000 7f7ab1b50700 1 -- 172.21.15.194:0/4010106428 shutdown_connections 2024-01-18T06:52:44.123 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:44.123+0000 7f7ab1b50700 1 --2- 172.21.15.194:0/4010106428 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f7a9402ff60 0x7f7a94032420 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:44.123 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:44.123+0000 7f7ab1b50700 1 --2- 172.21.15.194:0/4010106428 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7aac0f5410 0x7f7aac109450 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:44.123 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:44.123+0000 7f7ab1b50700 1 -- 172.21.15.194:0/4010106428 >> 172.21.15.194:0/4010106428 conn(0x7f7aac0f0b70 msgr2=0x7f7aac0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:52:44.123 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:44.123+0000 7f7ab1b50700 1 -- 172.21.15.194:0/4010106428 shutdown_connections 2024-01-18T06:52:44.123 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:44.123+0000 7f7ab1b50700 1 -- 172.21.15.194:0/4010106428 wait complete. 2024-01-18T06:52:44.123 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:52:45.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:45 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:52:44.013400+0000 mgr.smithi047.fuhzzr (mgr.14162) 128 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:52:45.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:45 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:52:44.122340+0000 mon.smithi047 (mon.0) 204 : audit [DBG] from='client.? 172.21.15.194:0/4010106428' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:52:45.535 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:52:45.536 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:52:46.717 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:46.716+0000 7faf19ddd700 1 -- 172.21.15.194:0/4198727110 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf14060220 msgr2=0x7faf140f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:46.718 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:46.716+0000 7faf19ddd700 1 --2- 172.21.15.194:0/4198727110 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf14060220 0x7faf140f9150 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7faf04009fb0 tx=0x7faf04009b30).stop 2024-01-18T06:52:46.718 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:46.716+0000 7faf19ddd700 1 -- 172.21.15.194:0/4198727110 shutdown_connections 2024-01-18T06:52:46.718 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:46.716+0000 7faf19ddd700 1 --2- 172.21.15.194:0/4198727110 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf14060220 0x7faf140f9150 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:46.718 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:46.716+0000 7faf19ddd700 1 -- 172.21.15.194:0/4198727110 >> 172.21.15.194:0/4198727110 conn(0x7faf140f0b70 msgr2=0x7faf140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:52:46.718 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:46.716+0000 7faf19ddd700 1 -- 172.21.15.194:0/4198727110 shutdown_connections 2024-01-18T06:52:46.718 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:46.716+0000 7faf19ddd700 1 -- 172.21.15.194:0/4198727110 wait complete. 2024-01-18T06:52:46.718 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:46.717+0000 7faf19ddd700 1 Processor -- start 2024-01-18T06:52:46.718 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:46.717+0000 7faf19ddd700 1 -- start start 2024-01-18T06:52:46.718 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:46.717+0000 7faf19ddd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf14060220 0x7faf141066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:52:46.718 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:46.717+0000 7faf19ddd700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf14106bd0 con 0x7faf14060220 2024-01-18T06:52:46.719 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:46.717+0000 7faf137fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf14060220 0x7faf141066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:52:46.719 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:46.717+0000 7faf137fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf14060220 0x7faf141066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:44704/0 (socket says 172.21.15.194:44704) 2024-01-18T06:52:46.719 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:46.717+0000 7faf137fe700 1 -- 172.21.15.194:0/3541360101 learned_addr learned my addr 172.21.15.194:0/3541360101 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:52:46.719 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:46.718+0000 7faf137fe700 1 -- 172.21.15.194:0/3541360101 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf04005040 con 0x7faf14060220 2024-01-18T06:52:46.719 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:46.718+0000 7faf137fe700 1 --2- 172.21.15.194:0/3541360101 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf14060220 0x7faf141066e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7faf04013510 tx=0x7faf0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:52:46.719 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:46.718+0000 7faf0bfff700 1 -- 172.21.15.194:0/3541360101 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf04009320 con 0x7faf14060220 2024-01-18T06:52:46.719 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:46.718+0000 7faf19ddd700 1 -- 172.21.15.194:0/3541360101 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf14106dd0 con 0x7faf14060220 2024-01-18T06:52:46.719 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:46.718+0000 7faf0bfff700 1 -- 172.21.15.194:0/3541360101 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faf04013f00 con 0x7faf14060220 2024-01-18T06:52:46.719 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:46.718+0000 7faf0bfff700 1 -- 172.21.15.194:0/3541360101 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf04014dc0 con 0x7faf14060220 2024-01-18T06:52:46.719 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:46.718+0000 7faf19ddd700 1 -- 172.21.15.194:0/3541360101 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf14107260 con 0x7faf14060220 2024-01-18T06:52:46.720 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:46.719+0000 7faf0bfff700 1 -- 172.21.15.194:0/3541360101 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faf0401d3d0 con 0x7faf14060220 2024-01-18T06:52:46.720 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:46.719+0000 7faf19ddd700 1 -- 172.21.15.194:0/3541360101 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf14045ad0 con 0x7faf14060220 2024-01-18T06:52:46.720 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:46.719+0000 7faf0bfff700 1 --2- 172.21.15.194:0/3541360101 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7faf0002ff60 0x7faf00032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:52:46.720 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:46.719+0000 7faf0bfff700 1 -- 172.21.15.194:0/3541360101 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faf040413c0 con 0x7faf14060220 2024-01-18T06:52:46.720 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:46.719+0000 7faf12ffd700 1 --2- 172.21.15.194:0/3541360101 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7faf0002ff60 0x7faf00032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:52:46.721 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:46.719+0000 7faf12ffd700 1 --2- 172.21.15.194:0/3541360101 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7faf0002ff60 0x7faf00032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7faefc0096a0 tx=0x7faefc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:52:46.723 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:46.723+0000 7faf0bfff700 1 -- 172.21.15.194:0/3541360101 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faf0401ccb0 con 0x7faf14060220 2024-01-18T06:52:46.930 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:46.930+0000 7faf19ddd700 1 -- 172.21.15.194:0/3541360101 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faf14108110 con 0x7faf14060220 2024-01-18T06:52:46.930 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:46.931+0000 7faf0bfff700 1 -- 172.21.15.194:0/3541360101 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7faf0401cad0 con 0x7faf14060220 2024-01-18T06:52:46.930 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:52:46.931 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:52:46.932 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:46.932+0000 7faf19ddd700 1 -- 172.21.15.194:0/3541360101 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7faf0002ff60 msgr2=0x7faf00032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:46.932 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:46.932+0000 7faf19ddd700 1 --2- 172.21.15.194:0/3541360101 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7faf0002ff60 0x7faf00032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7faefc0096a0 tx=0x7faefc006b40).stop 2024-01-18T06:52:46.932 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:46.932+0000 7faf19ddd700 1 -- 172.21.15.194:0/3541360101 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf14060220 msgr2=0x7faf141066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:46.932 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:46.932+0000 7faf19ddd700 1 --2- 172.21.15.194:0/3541360101 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf14060220 0x7faf141066e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7faf04013510 tx=0x7faf0400b040).stop 2024-01-18T06:52:46.932 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:46.932+0000 7faf19ddd700 1 -- 172.21.15.194:0/3541360101 shutdown_connections 2024-01-18T06:52:46.932 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:46.932+0000 7faf19ddd700 1 --2- 172.21.15.194:0/3541360101 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7faf0002ff60 0x7faf00032420 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:46.932 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:46.932+0000 7faf19ddd700 1 --2- 172.21.15.194:0/3541360101 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf14060220 0x7faf141066e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:46.932 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:46.932+0000 7faf19ddd700 1 -- 172.21.15.194:0/3541360101 >> 172.21.15.194:0/3541360101 conn(0x7faf140f0b70 msgr2=0x7faf140f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:52:46.932 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:46.932+0000 7faf19ddd700 1 -- 172.21.15.194:0/3541360101 shutdown_connections 2024-01-18T06:52:46.932 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:46.932+0000 7faf19ddd700 1 -- 172.21.15.194:0/3541360101 wait complete. 2024-01-18T06:52:46.933 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:52:47.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:47 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:52:46.013742+0000 mgr.smithi047.fuhzzr (mgr.14162) 129 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:52:47.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:47 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:52:46.931690+0000 mon.smithi047 (mon.0) 205 : audit [DBG] from='client.? 172.21.15.194:0/3541360101' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:52:48.343 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:52:48.343 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:52:49.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:48 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:52:48.014038+0000 mgr.smithi047.fuhzzr (mgr.14162) 130 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:52:49.531 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:49.531+0000 7fc8ed18e700 1 -- 172.21.15.194:0/1468405915 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8e80f5410 msgr2=0x7fc8e80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:49.531 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:49.531+0000 7fc8ed18e700 1 --2- 172.21.15.194:0/1468405915 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8e80f5410 0x7fc8e80f5830 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7fc8dc009fb0 tx=0x7fc8dc009b30).stop 2024-01-18T06:52:49.531 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:49.531+0000 7fc8ed18e700 1 -- 172.21.15.194:0/1468405915 shutdown_connections 2024-01-18T06:52:49.531 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:49.531+0000 7fc8ed18e700 1 --2- 172.21.15.194:0/1468405915 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8e80f5410 0x7fc8e80f5830 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:49.531 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:49.531+0000 7fc8ed18e700 1 -- 172.21.15.194:0/1468405915 >> 172.21.15.194:0/1468405915 conn(0x7fc8e80f0b70 msgr2=0x7fc8e80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:52:49.531 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:49.531+0000 7fc8ed18e700 1 -- 172.21.15.194:0/1468405915 shutdown_connections 2024-01-18T06:52:49.532 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:49.531+0000 7fc8ed18e700 1 -- 172.21.15.194:0/1468405915 wait complete. 2024-01-18T06:52:49.532 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:49.532+0000 7fc8ed18e700 1 Processor -- start 2024-01-18T06:52:49.532 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:49.532+0000 7fc8ed18e700 1 -- start start 2024-01-18T06:52:49.534 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:49.532+0000 7fc8ed18e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8e80f5410 0x7fc8e8109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:52:49.534 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:49.532+0000 7fc8ed18e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e8060240 con 0x7fc8e80f5410 2024-01-18T06:52:49.534 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:49.532+0000 7fc8e6d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8e80f5410 0x7fc8e8109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:52:49.534 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:49.532+0000 7fc8e6d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8e80f5410 0x7fc8e8109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:38768/0 (socket says 172.21.15.194:38768) 2024-01-18T06:52:49.534 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:49.532+0000 7fc8e6d9d700 1 -- 172.21.15.194:0/2587737201 learned_addr learned my addr 172.21.15.194:0/2587737201 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:52:49.534 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:49.533+0000 7fc8e6d9d700 1 -- 172.21.15.194:0/2587737201 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8dc005040 con 0x7fc8e80f5410 2024-01-18T06:52:49.534 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:49.533+0000 7fc8e6d9d700 1 --2- 172.21.15.194:0/2587737201 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8e80f5410 0x7fc8e8109430 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7fc8dc013510 tx=0x7fc8dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:52:49.534 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:49.533+0000 7fc8cf7fe700 1 -- 172.21.15.194:0/2587737201 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc8dc009320 con 0x7fc8e80f5410 2024-01-18T06:52:49.534 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:49.533+0000 7fc8cf7fe700 1 -- 172.21.15.194:0/2587737201 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc8dc013f00 con 0x7fc8e80f5410 2024-01-18T06:52:49.534 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:49.533+0000 7fc8ed18e700 1 -- 172.21.15.194:0/2587737201 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8e8060440 con 0x7fc8e80f5410 2024-01-18T06:52:49.534 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:49.533+0000 7fc8cf7fe700 1 -- 172.21.15.194:0/2587737201 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc8dc014d90 con 0x7fc8e80f5410 2024-01-18T06:52:49.534 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:49.533+0000 7fc8ed18e700 1 -- 172.21.15.194:0/2587737201 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8e8109ae0 con 0x7fc8e80f5410 2024-01-18T06:52:49.534 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:49.534+0000 7fc8cf7fe700 1 -- 172.21.15.194:0/2587737201 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc8dc0263a0 con 0x7fc8e80f5410 2024-01-18T06:52:49.535 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:49.534+0000 7fc8ed18e700 1 -- 172.21.15.194:0/2587737201 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8e8045ad0 con 0x7fc8e80f5410 2024-01-18T06:52:49.535 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:49.534+0000 7fc8cf7fe700 1 --2- 172.21.15.194:0/2587737201 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc8d002ffb0 0x7fc8d0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:52:49.535 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:49.534+0000 7fc8cf7fe700 1 -- 172.21.15.194:0/2587737201 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc8dc0412b0 con 0x7fc8e80f5410 2024-01-18T06:52:49.535 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:49.534+0000 7fc8e659c700 1 --2- 172.21.15.194:0/2587737201 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc8d002ffb0 0x7fc8d0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:52:49.535 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:49.534+0000 7fc8e659c700 1 --2- 172.21.15.194:0/2587737201 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc8d002ffb0 0x7fc8d0032470 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fc8d80096a0 tx=0x7fc8d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:52:49.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:49.539+0000 7fc8cf7fe700 1 -- 172.21.15.194:0/2587737201 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc8dc01ccf0 con 0x7fc8e80f5410 2024-01-18T06:52:49.744 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:49.744+0000 7fc8ed18e700 1 -- 172.21.15.194:0/2587737201 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc8e80fae60 con 0x7fc8e80f5410 2024-01-18T06:52:49.744 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:49.745+0000 7fc8cf7fe700 1 -- 172.21.15.194:0/2587737201 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc8dc01cb10 con 0x7fc8e80f5410 2024-01-18T06:52:49.745 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:52:49.745 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:52:49.746 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:49.747+0000 7fc8ed18e700 1 -- 172.21.15.194:0/2587737201 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc8d002ffb0 msgr2=0x7fc8d0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:49.746 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:49.747+0000 7fc8ed18e700 1 --2- 172.21.15.194:0/2587737201 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc8d002ffb0 0x7fc8d0032470 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fc8d80096a0 tx=0x7fc8d8006b40).stop 2024-01-18T06:52:49.746 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:49.747+0000 7fc8ed18e700 1 -- 172.21.15.194:0/2587737201 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8e80f5410 msgr2=0x7fc8e8109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:49.747 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:49.747+0000 7fc8ed18e700 1 --2- 172.21.15.194:0/2587737201 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8e80f5410 0x7fc8e8109430 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7fc8dc013510 tx=0x7fc8dc00b040).stop 2024-01-18T06:52:49.747 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:49.747+0000 7fc8ed18e700 1 -- 172.21.15.194:0/2587737201 shutdown_connections 2024-01-18T06:52:49.747 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:49.747+0000 7fc8ed18e700 1 --2- 172.21.15.194:0/2587737201 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc8d002ffb0 0x7fc8d0032470 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:49.747 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:49.747+0000 7fc8ed18e700 1 --2- 172.21.15.194:0/2587737201 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8e80f5410 0x7fc8e8109430 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:49.747 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:49.747+0000 7fc8ed18e700 1 -- 172.21.15.194:0/2587737201 >> 172.21.15.194:0/2587737201 conn(0x7fc8e80f0b70 msgr2=0x7fc8e80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:52:49.747 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:49.747+0000 7fc8ed18e700 1 -- 172.21.15.194:0/2587737201 shutdown_connections 2024-01-18T06:52:49.747 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:49.747+0000 7fc8ed18e700 1 -- 172.21.15.194:0/2587737201 wait complete. 2024-01-18T06:52:49.747 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:52:50.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:49 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:52:49.745844+0000 mon.smithi047 (mon.0) 206 : audit [DBG] from='client.? 172.21.15.194:0/2587737201' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:52:51.160 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:52:51.160 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:52:51.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:50 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:52:50.014312+0000 mgr.smithi047.fuhzzr (mgr.14162) 131 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:52:52.295 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:52.294+0000 7f56bd3aa700 1 -- 172.21.15.194:0/1361986379 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f56b80f6430 msgr2=0x7f56b80f6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:52.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:52.294+0000 7f56bd3aa700 1 --2- 172.21.15.194:0/1361986379 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f56b80f6430 0x7f56b80f6850 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f56a8009fb0 tx=0x7f56a8009b30).stop 2024-01-18T06:52:52.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:52.294+0000 7f56bd3aa700 1 -- 172.21.15.194:0/1361986379 shutdown_connections 2024-01-18T06:52:52.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:52.294+0000 7f56bd3aa700 1 --2- 172.21.15.194:0/1361986379 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f56b80f6430 0x7f56b80f6850 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:52.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:52.294+0000 7f56bd3aa700 1 -- 172.21.15.194:0/1361986379 >> 172.21.15.194:0/1361986379 conn(0x7f56b80f1b90 msgr2=0x7f56b80f3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:52:52.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:52.294+0000 7f56bd3aa700 1 -- 172.21.15.194:0/1361986379 shutdown_connections 2024-01-18T06:52:52.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:52.294+0000 7f56bd3aa700 1 -- 172.21.15.194:0/1361986379 wait complete. 2024-01-18T06:52:52.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:52.295+0000 7f56bd3aa700 1 Processor -- start 2024-01-18T06:52:52.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:52.295+0000 7f56bd3aa700 1 -- start start 2024-01-18T06:52:52.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:52.295+0000 7f56bd3aa700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f56b80f6430 0x7f56b8106650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:52:52.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:52.295+0000 7f56bd3aa700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b8106b40 con 0x7f56b80f6430 2024-01-18T06:52:52.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:52.295+0000 7f56b6ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f56b80f6430 0x7f56b8106650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:52:52.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:52.296+0000 7f56b6ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f56b80f6430 0x7f56b8106650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:38792/0 (socket says 172.21.15.194:38792) 2024-01-18T06:52:52.297 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:52.296+0000 7f56b6ffd700 1 -- 172.21.15.194:0/468862238 learned_addr learned my addr 172.21.15.194:0/468862238 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:52:52.297 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:52.296+0000 7f56b6ffd700 1 -- 172.21.15.194:0/468862238 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a8005040 con 0x7f56b80f6430 2024-01-18T06:52:52.297 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:52.296+0000 7f56b6ffd700 1 --2- 172.21.15.194:0/468862238 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f56b80f6430 0x7f56b8106650 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f56a8014000 tx=0x7f56a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:52:52.297 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:52.296+0000 7f569f7fe700 1 -- 172.21.15.194:0/468862238 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56a80095f0 con 0x7f56b80f6430 2024-01-18T06:52:52.297 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:52.296+0000 7f56bd3aa700 1 -- 172.21.15.194:0/468862238 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56b8106d40 con 0x7f56b80f6430 2024-01-18T06:52:52.297 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:52.296+0000 7f569f7fe700 1 -- 172.21.15.194:0/468862238 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f56a8007e70 con 0x7f56b80f6430 2024-01-18T06:52:52.297 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:52.296+0000 7f569f7fe700 1 -- 172.21.15.194:0/468862238 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56a8014ea0 con 0x7f56b80f6430 2024-01-18T06:52:52.297 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:52.296+0000 7f56bd3aa700 1 -- 172.21.15.194:0/468862238 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56b81071d0 con 0x7f56b80f6430 2024-01-18T06:52:52.298 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:52.297+0000 7f569f7fe700 1 -- 172.21.15.194:0/468862238 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f56a8025070 con 0x7f56b80f6430 2024-01-18T06:52:52.298 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:52.297+0000 7f569f7fe700 1 --2- 172.21.15.194:0/468862238 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f56a002ff60 0x7f56a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:52:52.298 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:52.297+0000 7f569f7fe700 1 -- 172.21.15.194:0/468862238 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f56a8040a20 con 0x7f56b80f6430 2024-01-18T06:52:52.298 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:52.297+0000 7f56bd3aa700 1 -- 172.21.15.194:0/468862238 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56b8045ad0 con 0x7f56b80f6430 2024-01-18T06:52:52.298 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:52.297+0000 7f56b67fc700 1 --2- 172.21.15.194:0/468862238 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f56a002ff60 0x7f56a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:52:52.298 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:52.298+0000 7f56b67fc700 1 --2- 172.21.15.194:0/468862238 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f56a002ff60 0x7f56a0032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f56ac0096a0 tx=0x7f56ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:52:52.301 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:52.302+0000 7f569f7fe700 1 -- 172.21.15.194:0/468862238 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f56a8026350 con 0x7f56b80f6430 2024-01-18T06:52:52.510 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:52.510+0000 7f56bd3aa700 1 -- 172.21.15.194:0/468862238 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f56b81081e0 con 0x7f56b80f6430 2024-01-18T06:52:52.510 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:52.511+0000 7f569f7fe700 1 -- 172.21.15.194:0/468862238 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f56a802d090 con 0x7f56b80f6430 2024-01-18T06:52:52.511 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:52:52.511 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:52:52.512 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:52.513+0000 7f56bd3aa700 1 -- 172.21.15.194:0/468862238 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f56a002ff60 msgr2=0x7f56a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:52.512 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:52.513+0000 7f56bd3aa700 1 --2- 172.21.15.194:0/468862238 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f56a002ff60 0x7f56a0032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f56ac0096a0 tx=0x7f56ac006b40).stop 2024-01-18T06:52:52.512 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:52.513+0000 7f56bd3aa700 1 -- 172.21.15.194:0/468862238 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f56b80f6430 msgr2=0x7f56b8106650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:52.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:52.513+0000 7f56bd3aa700 1 --2- 172.21.15.194:0/468862238 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f56b80f6430 0x7f56b8106650 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f56a8014000 tx=0x7f56a800b040).stop 2024-01-18T06:52:52.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:52.513+0000 7f56bd3aa700 1 -- 172.21.15.194:0/468862238 shutdown_connections 2024-01-18T06:52:52.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:52.513+0000 7f56bd3aa700 1 --2- 172.21.15.194:0/468862238 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f56a002ff60 0x7f56a0032420 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:52.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:52.513+0000 7f56bd3aa700 1 --2- 172.21.15.194:0/468862238 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f56b80f6430 0x7f56b8106650 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:52.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:52.513+0000 7f56bd3aa700 1 -- 172.21.15.194:0/468862238 >> 172.21.15.194:0/468862238 conn(0x7f56b80f1b90 msgr2=0x7f56b80f2630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:52:52.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:52.513+0000 7f56bd3aa700 1 -- 172.21.15.194:0/468862238 shutdown_connections 2024-01-18T06:52:52.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:52.513+0000 7f56bd3aa700 1 -- 172.21.15.194:0/468862238 wait complete. 2024-01-18T06:52:52.513 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:52:53.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:53 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:52:52.014665+0000 mgr.smithi047.fuhzzr (mgr.14162) 132 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:52:53.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:53 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:52:52.512044+0000 mon.smithi047 (mon.0) 207 : audit [DBG] from='client.? 172.21.15.194:0/468862238' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:52:53.928 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:52:53.928 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:52:55.107 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:55.105+0000 7f292d823700 1 -- 172.21.15.194:0/2813686082 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f29280f4800 msgr2=0x7f29280f4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:55.107 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:55.105+0000 7f292d823700 1 --2- 172.21.15.194:0/2813686082 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f29280f4800 0x7f29280f4c20 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f291c009fb0 tx=0x7f291c009b30).stop 2024-01-18T06:52:55.107 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:55.105+0000 7f292d823700 1 -- 172.21.15.194:0/2813686082 shutdown_connections 2024-01-18T06:52:55.107 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:55.105+0000 7f292d823700 1 --2- 172.21.15.194:0/2813686082 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f29280f4800 0x7f29280f4c20 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:55.107 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:55.105+0000 7f292d823700 1 -- 172.21.15.194:0/2813686082 >> 172.21.15.194:0/2813686082 conn(0x7f29280f0260 msgr2=0x7f29280f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:52:55.107 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:55.105+0000 7f292d823700 1 -- 172.21.15.194:0/2813686082 shutdown_connections 2024-01-18T06:52:55.107 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:55.106+0000 7f292d823700 1 -- 172.21.15.194:0/2813686082 wait complete. 2024-01-18T06:52:55.108 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:55.106+0000 7f292d823700 1 Processor -- start 2024-01-18T06:52:55.108 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:55.106+0000 7f292d823700 1 -- start start 2024-01-18T06:52:55.108 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:55.106+0000 7f292d823700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f29280f4800 0x7f29281066b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:52:55.108 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:55.106+0000 7f292d823700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2928106ba0 con 0x7f29280f4800 2024-01-18T06:52:55.109 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:55.107+0000 7f2926ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f29280f4800 0x7f29281066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:52:55.109 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:55.107+0000 7f2926ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f29280f4800 0x7f29281066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:38810/0 (socket says 172.21.15.194:38810) 2024-01-18T06:52:55.109 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:55.107+0000 7f2926ffd700 1 -- 172.21.15.194:0/501721207 learned_addr learned my addr 172.21.15.194:0/501721207 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:52:55.109 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:55.107+0000 7f2926ffd700 1 -- 172.21.15.194:0/501721207 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f291c005040 con 0x7f29280f4800 2024-01-18T06:52:55.109 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:55.107+0000 7f2926ffd700 1 --2- 172.21.15.194:0/501721207 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f29280f4800 0x7f29281066b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f291c013510 tx=0x7f291c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:52:55.109 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:55.107+0000 7f290f7fe700 1 -- 172.21.15.194:0/501721207 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f291c009320 con 0x7f29280f4800 2024-01-18T06:52:55.109 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:55.107+0000 7f290f7fe700 1 -- 172.21.15.194:0/501721207 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f291c013f00 con 0x7f29280f4800 2024-01-18T06:52:55.109 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:55.107+0000 7f292d823700 1 -- 172.21.15.194:0/501721207 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2928106da0 con 0x7f29280f4800 2024-01-18T06:52:55.109 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:55.107+0000 7f290f7fe700 1 -- 172.21.15.194:0/501721207 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f291c014dc0 con 0x7f29280f4800 2024-01-18T06:52:55.109 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:55.107+0000 7f292d823700 1 -- 172.21.15.194:0/501721207 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2928107230 con 0x7f29280f4800 2024-01-18T06:52:55.110 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:55.108+0000 7f290f7fe700 1 -- 172.21.15.194:0/501721207 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f291c01d3d0 con 0x7f29280f4800 2024-01-18T06:52:55.110 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:55.108+0000 7f290f7fe700 1 --2- 172.21.15.194:0/501721207 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f291002ffb0 0x7f2910032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:52:55.110 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:55.108+0000 7f292d823700 1 -- 172.21.15.194:0/501721207 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2928045ad0 con 0x7f29280f4800 2024-01-18T06:52:55.110 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:55.108+0000 7f290f7fe700 1 -- 172.21.15.194:0/501721207 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f291c0410e0 con 0x7f29280f4800 2024-01-18T06:52:55.110 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:55.109+0000 7f29267fc700 1 --2- 172.21.15.194:0/501721207 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f291002ffb0 0x7f2910032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:52:55.111 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:55.109+0000 7f29267fc700 1 --2- 172.21.15.194:0/501721207 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f291002ffb0 0x7f2910032470 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f29180096a0 tx=0x7f2918006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:52:55.112 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:55.113+0000 7f290f7fe700 1 -- 172.21.15.194:0/501721207 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f291c012920 con 0x7f29280f4800 2024-01-18T06:52:55.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:55.317+0000 7f292d823700 1 -- 172.21.15.194:0/501721207 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f29281080e0 con 0x7f29280f4800 2024-01-18T06:52:55.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:55.317+0000 7f290f7fe700 1 -- 172.21.15.194:0/501721207 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f291c03a020 con 0x7f29280f4800 2024-01-18T06:52:55.317 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:52:55.317 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:52:55.318 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:55.319+0000 7f292d823700 1 -- 172.21.15.194:0/501721207 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f291002ffb0 msgr2=0x7f2910032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:55.318 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:55.319+0000 7f292d823700 1 --2- 172.21.15.194:0/501721207 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f291002ffb0 0x7f2910032470 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f29180096a0 tx=0x7f2918006b40).stop 2024-01-18T06:52:55.319 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:55.319+0000 7f292d823700 1 -- 172.21.15.194:0/501721207 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f29280f4800 msgr2=0x7f29281066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:55.319 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:55.319+0000 7f292d823700 1 --2- 172.21.15.194:0/501721207 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f29280f4800 0x7f29281066b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f291c013510 tx=0x7f291c00b040).stop 2024-01-18T06:52:55.319 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:55.319+0000 7f292d823700 1 -- 172.21.15.194:0/501721207 shutdown_connections 2024-01-18T06:52:55.319 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:55.319+0000 7f292d823700 1 --2- 172.21.15.194:0/501721207 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f291002ffb0 0x7f2910032470 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:55.319 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:55.319+0000 7f292d823700 1 --2- 172.21.15.194:0/501721207 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f29280f4800 0x7f29281066b0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:55.319 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:55.319+0000 7f292d823700 1 -- 172.21.15.194:0/501721207 >> 172.21.15.194:0/501721207 conn(0x7f29280f0260 msgr2=0x7f29280f0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:52:55.319 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:55.319+0000 7f292d823700 1 -- 172.21.15.194:0/501721207 shutdown_connections 2024-01-18T06:52:55.319 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:55.319+0000 7f292d823700 1 -- 172.21.15.194:0/501721207 wait complete. 2024-01-18T06:52:55.319 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:52:55.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:55 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:52:54.014942+0000 mgr.smithi047.fuhzzr (mgr.14162) 133 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:52:56.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:52:55.318121+0000 mon.smithi047 (mon.0) 208 : audit [DBG] from='client.? 172.21.15.194:0/501721207' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:52:56.726 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:52:56.726 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:52:57.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:57 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:52:56.015270+0000 mgr.smithi047.fuhzzr (mgr.14162) 134 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:52:57.880 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:57.880+0000 7f782759e700 1 -- 172.21.15.194:0/275464384 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f78280f62a0 msgr2=0x7f78280f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:57.880 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:57.880+0000 7f782759e700 1 --2- 172.21.15.194:0/275464384 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f78280f62a0 0x7f78280f66c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f7818009fb0 tx=0x7f7818009b30).stop 2024-01-18T06:52:57.880 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:57.880+0000 7f782759e700 1 -- 172.21.15.194:0/275464384 shutdown_connections 2024-01-18T06:52:57.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:57.880+0000 7f782759e700 1 --2- 172.21.15.194:0/275464384 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f78280f62a0 0x7f78280f66c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:57.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:57.880+0000 7f782759e700 1 -- 172.21.15.194:0/275464384 >> 172.21.15.194:0/275464384 conn(0x7f78280f1aa0 msgr2=0x7f78280f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:52:57.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:57.880+0000 7f782759e700 1 -- 172.21.15.194:0/275464384 shutdown_connections 2024-01-18T06:52:57.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:57.880+0000 7f782759e700 1 -- 172.21.15.194:0/275464384 wait complete. 2024-01-18T06:52:57.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:57.880+0000 7f782759e700 1 Processor -- start 2024-01-18T06:52:57.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:57.880+0000 7f782759e700 1 -- start start 2024-01-18T06:52:57.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:57.880+0000 7f782759e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f78280f62a0 0x7f78281064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:52:57.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:57.880+0000 7f782759e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78281069a0 con 0x7f78280f62a0 2024-01-18T06:52:57.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:57.881+0000 7f782659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f78280f62a0 0x7f78281064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:52:57.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:57.881+0000 7f782659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f78280f62a0 0x7f78281064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:40954/0 (socket says 172.21.15.194:40954) 2024-01-18T06:52:57.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:57.881+0000 7f782659c700 1 -- 172.21.15.194:0/1771949876 learned_addr learned my addr 172.21.15.194:0/1771949876 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:52:57.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:57.881+0000 7f782659c700 1 -- 172.21.15.194:0/1771949876 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7818005040 con 0x7f78280f62a0 2024-01-18T06:52:57.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:57.881+0000 7f782659c700 1 --2- 172.21.15.194:0/1771949876 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f78280f62a0 0x7f78281064b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f7818013510 tx=0x7f781800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:52:57.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:57.881+0000 7f7816ffd700 1 -- 172.21.15.194:0/1771949876 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7818009320 con 0x7f78280f62a0 2024-01-18T06:52:57.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:57.881+0000 7f782759e700 1 -- 172.21.15.194:0/1771949876 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7828106ba0 con 0x7f78280f62a0 2024-01-18T06:52:57.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:57.881+0000 7f7816ffd700 1 -- 172.21.15.194:0/1771949876 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7818013f00 con 0x7f78280f62a0 2024-01-18T06:52:57.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:57.881+0000 7f7816ffd700 1 -- 172.21.15.194:0/1771949876 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7818014dc0 con 0x7f78280f62a0 2024-01-18T06:52:57.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:57.881+0000 7f782759e700 1 -- 172.21.15.194:0/1771949876 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7828107030 con 0x7f78280f62a0 2024-01-18T06:52:57.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:57.882+0000 7f7816ffd700 1 -- 172.21.15.194:0/1771949876 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f78180095f0 con 0x7f78280f62a0 2024-01-18T06:52:57.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:57.882+0000 7f782759e700 1 -- 172.21.15.194:0/1771949876 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7828045ad0 con 0x7f78280f62a0 2024-01-18T06:52:57.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:57.882+0000 7f7816ffd700 1 --2- 172.21.15.194:0/1771949876 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f781002ff60 0x7f7810032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:52:57.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:57.882+0000 7f7816ffd700 1 -- 172.21.15.194:0/1771949876 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7818041020 con 0x7f78280f62a0 2024-01-18T06:52:57.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:57.882+0000 7f7825d9b700 1 --2- 172.21.15.194:0/1771949876 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f781002ff60 0x7f7810032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:52:57.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:57.883+0000 7f7825d9b700 1 --2- 172.21.15.194:0/1771949876 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f781002ff60 0x7f7810032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f781c0096a0 tx=0x7f781c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:52:57.886 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:57.887+0000 7f7816ffd700 1 -- 172.21.15.194:0/1771949876 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7818014420 con 0x7f78280f62a0 2024-01-18T06:52:58.092 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:58.093+0000 7f782759e700 1 -- 172.21.15.194:0/1771949876 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7828107e80 con 0x7f78280f62a0 2024-01-18T06:52:58.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:58.094+0000 7f7816ffd700 1 -- 172.21.15.194:0/1771949876 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f781801c920 con 0x7f78280f62a0 2024-01-18T06:52:58.094 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:52:58.094 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:52:58.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:58.095+0000 7f782759e700 1 -- 172.21.15.194:0/1771949876 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f781002ff60 msgr2=0x7f7810032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:58.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:58.095+0000 7f782759e700 1 --2- 172.21.15.194:0/1771949876 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f781002ff60 0x7f7810032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f781c0096a0 tx=0x7f781c006b40).stop 2024-01-18T06:52:58.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:58.095+0000 7f782759e700 1 -- 172.21.15.194:0/1771949876 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f78280f62a0 msgr2=0x7f78281064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:52:58.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:58.095+0000 7f782759e700 1 --2- 172.21.15.194:0/1771949876 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f78280f62a0 0x7f78281064b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f7818013510 tx=0x7f781800b040).stop 2024-01-18T06:52:58.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:58.095+0000 7f782759e700 1 -- 172.21.15.194:0/1771949876 shutdown_connections 2024-01-18T06:52:58.096 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:58.095+0000 7f782759e700 1 --2- 172.21.15.194:0/1771949876 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f781002ff60 0x7f7810032420 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:58.096 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:58.095+0000 7f782759e700 1 --2- 172.21.15.194:0/1771949876 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f78280f62a0 0x7f78281064b0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:52:58.096 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:58.095+0000 7f782759e700 1 -- 172.21.15.194:0/1771949876 >> 172.21.15.194:0/1771949876 conn(0x7f78280f1aa0 msgr2=0x7f78280f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:52:58.096 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:58.096+0000 7f782759e700 1 -- 172.21.15.194:0/1771949876 shutdown_connections 2024-01-18T06:52:58.096 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:52:58.096+0000 7f782759e700 1 -- 172.21.15.194:0/1771949876 wait complete. 2024-01-18T06:52:58.096 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:52:59.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:58 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:52:58.015654+0000 mgr.smithi047.fuhzzr (mgr.14162) 135 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:52:59.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:52:58 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:52:58.094763+0000 mon.smithi047 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.194:0/1771949876' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:52:59.485 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:52:59.485 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:53:00.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:00.635+0000 7f66706c5700 1 -- 172.21.15.194:0/757109833 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66680f6510 msgr2=0x7f66680f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:00.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:00.635+0000 7f66706c5700 1 --2- 172.21.15.194:0/757109833 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66680f6510 0x7f66680f6930 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f6658009fb0 tx=0x7f6658009b30).stop 2024-01-18T06:53:00.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:00.636+0000 7f66706c5700 1 -- 172.21.15.194:0/757109833 shutdown_connections 2024-01-18T06:53:00.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:00.636+0000 7f66706c5700 1 --2- 172.21.15.194:0/757109833 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66680f6510 0x7f66680f6930 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:00.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:00.636+0000 7f66706c5700 1 -- 172.21.15.194:0/757109833 >> 172.21.15.194:0/757109833 conn(0x7f66680f1c70 msgr2=0x7f66680f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:00.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:00.636+0000 7f66706c5700 1 -- 172.21.15.194:0/757109833 shutdown_connections 2024-01-18T06:53:00.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:00.636+0000 7f66706c5700 1 -- 172.21.15.194:0/757109833 wait complete. 2024-01-18T06:53:00.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:00.636+0000 7f66706c5700 1 Processor -- start 2024-01-18T06:53:00.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:00.636+0000 7f66706c5700 1 -- start start 2024-01-18T06:53:00.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:00.636+0000 7f66706c5700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66680f6510 0x7f6668106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:00.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:00.636+0000 7f66706c5700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6668106c20 con 0x7f66680f6510 2024-01-18T06:53:00.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:00.636+0000 7f666e461700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66680f6510 0x7f6668106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:00.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:00.637+0000 7f666e461700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66680f6510 0x7f6668106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:40962/0 (socket says 172.21.15.194:40962) 2024-01-18T06:53:00.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:00.637+0000 7f666e461700 1 -- 172.21.15.194:0/1696044274 learned_addr learned my addr 172.21.15.194:0/1696044274 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:53:00.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:00.637+0000 7f666e461700 1 -- 172.21.15.194:0/1696044274 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6658005040 con 0x7f66680f6510 2024-01-18T06:53:00.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:00.637+0000 7f666e461700 1 --2- 172.21.15.194:0/1696044274 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66680f6510 0x7f6668106730 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f6658013510 tx=0x7f665800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:00.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:00.637+0000 7f665effd700 1 -- 172.21.15.194:0/1696044274 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6658009320 con 0x7f66680f6510 2024-01-18T06:53:00.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:00.637+0000 7f66706c5700 1 -- 172.21.15.194:0/1696044274 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6668106e20 con 0x7f66680f6510 2024-01-18T06:53:00.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:00.637+0000 7f665effd700 1 -- 172.21.15.194:0/1696044274 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6658013f00 con 0x7f66680f6510 2024-01-18T06:53:00.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:00.637+0000 7f665effd700 1 -- 172.21.15.194:0/1696044274 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6658014d90 con 0x7f66680f6510 2024-01-18T06:53:00.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:00.637+0000 7f66706c5700 1 -- 172.21.15.194:0/1696044274 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66681072b0 con 0x7f66680f6510 2024-01-18T06:53:00.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:00.638+0000 7f66706c5700 1 -- 172.21.15.194:0/1696044274 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6668045ad0 con 0x7f66680f6510 2024-01-18T06:53:00.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:00.638+0000 7f665effd700 1 -- 172.21.15.194:0/1696044274 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f66580095f0 con 0x7f66680f6510 2024-01-18T06:53:00.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:00.638+0000 7f665effd700 1 --2- 172.21.15.194:0/1696044274 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f665402ff60 0x7f6654032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:00.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:00.638+0000 7f665effd700 1 -- 172.21.15.194:0/1696044274 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6658041000 con 0x7f66680f6510 2024-01-18T06:53:00.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:00.638+0000 7f666dc60700 1 --2- 172.21.15.194:0/1696044274 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f665402ff60 0x7f6654032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:00.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:00.639+0000 7f666dc60700 1 --2- 172.21.15.194:0/1696044274 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f665402ff60 0x7f6654032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f66640096a0 tx=0x7f6664006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:00.642 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:00.643+0000 7f665effd700 1 -- 172.21.15.194:0/1696044274 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6658014420 con 0x7f66680f6510 2024-01-18T06:53:00.850 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:00.850+0000 7f66706c5700 1 -- 172.21.15.194:0/1696044274 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f66680fbf60 con 0x7f66680f6510 2024-01-18T06:53:00.851 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:00.850+0000 7f665effd700 1 -- 172.21.15.194:0/1696044274 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6658010230 con 0x7f66680f6510 2024-01-18T06:53:00.851 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:53:00.851 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:53:00.853 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:00.852+0000 7f66706c5700 1 -- 172.21.15.194:0/1696044274 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f665402ff60 msgr2=0x7f6654032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:00.853 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:00.852+0000 7f66706c5700 1 --2- 172.21.15.194:0/1696044274 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f665402ff60 0x7f6654032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f66640096a0 tx=0x7f6664006b40).stop 2024-01-18T06:53:00.853 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:00.852+0000 7f66706c5700 1 -- 172.21.15.194:0/1696044274 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66680f6510 msgr2=0x7f6668106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:00.853 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:00.852+0000 7f66706c5700 1 --2- 172.21.15.194:0/1696044274 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66680f6510 0x7f6668106730 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f6658013510 tx=0x7f665800b040).stop 2024-01-18T06:53:00.853 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:00.852+0000 7f66706c5700 1 -- 172.21.15.194:0/1696044274 shutdown_connections 2024-01-18T06:53:00.853 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:00.852+0000 7f66706c5700 1 --2- 172.21.15.194:0/1696044274 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f665402ff60 0x7f6654032420 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:00.853 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:00.852+0000 7f66706c5700 1 --2- 172.21.15.194:0/1696044274 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66680f6510 0x7f6668106730 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:00.853 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:00.852+0000 7f66706c5700 1 -- 172.21.15.194:0/1696044274 >> 172.21.15.194:0/1696044274 conn(0x7f66680f1c70 msgr2=0x7f66680f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:00.853 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:00.852+0000 7f66706c5700 1 -- 172.21.15.194:0/1696044274 shutdown_connections 2024-01-18T06:53:00.853 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:00.852+0000 7f66706c5700 1 -- 172.21.15.194:0/1696044274 wait complete. 2024-01-18T06:53:00.854 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:53:01.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:01 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:53:00.015961+0000 mgr.smithi047.fuhzzr (mgr.14162) 136 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:53:01.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:01 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:53:00.852423+0000 mon.smithi047 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.194:0/1696044274' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:53:02.264 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:53:02.264 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:53:03.459 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:03.458+0000 7fb2bd520700 1 -- 172.21.15.194:0/3621118855 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb2b80f5410 msgr2=0x7fb2b80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:03.459 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:03.458+0000 7fb2bd520700 1 --2- 172.21.15.194:0/3621118855 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb2b80f5410 0x7fb2b80f5830 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7fb2ac009fb0 tx=0x7fb2ac009b30).stop 2024-01-18T06:53:03.459 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:03.458+0000 7fb2bd520700 1 -- 172.21.15.194:0/3621118855 shutdown_connections 2024-01-18T06:53:03.459 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:03.458+0000 7fb2bd520700 1 --2- 172.21.15.194:0/3621118855 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb2b80f5410 0x7fb2b80f5830 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:03.459 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:03.458+0000 7fb2bd520700 1 -- 172.21.15.194:0/3621118855 >> 172.21.15.194:0/3621118855 conn(0x7fb2b80f0b70 msgr2=0x7fb2b80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:03.459 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:03.458+0000 7fb2bd520700 1 -- 172.21.15.194:0/3621118855 shutdown_connections 2024-01-18T06:53:03.459 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:03.458+0000 7fb2bd520700 1 -- 172.21.15.194:0/3621118855 wait complete. 2024-01-18T06:53:03.461 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:03.459+0000 7fb2bd520700 1 Processor -- start 2024-01-18T06:53:03.462 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:03.459+0000 7fb2bd520700 1 -- start start 2024-01-18T06:53:03.462 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:03.459+0000 7fb2bd520700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb2b80f5410 0x7fb2b80fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:03.462 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:03.459+0000 7fb2bd520700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b80fb820 con 0x7fb2b80f5410 2024-01-18T06:53:03.462 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:03.459+0000 7fb2b6ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb2b80f5410 0x7fb2b80fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:03.462 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:03.459+0000 7fb2b6ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb2b80f5410 0x7fb2b80fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:40976/0 (socket says 172.21.15.194:40976) 2024-01-18T06:53:03.462 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:03.459+0000 7fb2b6ffd700 1 -- 172.21.15.194:0/1263575989 learned_addr learned my addr 172.21.15.194:0/1263575989 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:53:03.462 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:03.460+0000 7fb2b6ffd700 1 -- 172.21.15.194:0/1263575989 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2ac005040 con 0x7fb2b80f5410 2024-01-18T06:53:03.462 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:03.460+0000 7fb2b6ffd700 1 --2- 172.21.15.194:0/1263575989 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb2b80f5410 0x7fb2b80fd3a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7fb2ac013510 tx=0x7fb2ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:03.462 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:03.460+0000 7fb29f7fe700 1 -- 172.21.15.194:0/1263575989 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2ac009320 con 0x7fb2b80f5410 2024-01-18T06:53:03.462 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:03.460+0000 7fb2bd520700 1 -- 172.21.15.194:0/1263575989 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2b80fb9c0 con 0x7fb2b80f5410 2024-01-18T06:53:03.462 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:03.460+0000 7fb29f7fe700 1 -- 172.21.15.194:0/1263575989 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb2ac013f00 con 0x7fb2b80f5410 2024-01-18T06:53:03.462 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:03.460+0000 7fb29f7fe700 1 -- 172.21.15.194:0/1263575989 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2ac014d90 con 0x7fb2b80f5410 2024-01-18T06:53:03.462 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:03.460+0000 7fb2bd520700 1 -- 172.21.15.194:0/1263575989 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2b80fbe50 con 0x7fb2b80f5410 2024-01-18T06:53:03.462 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:03.461+0000 7fb2bd520700 1 -- 172.21.15.194:0/1263575989 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2b8059100 con 0x7fb2b80f5410 2024-01-18T06:53:03.463 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:03.461+0000 7fb29f7fe700 1 -- 172.21.15.194:0/1263575989 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb2ac014420 con 0x7fb2b80f5410 2024-01-18T06:53:03.463 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:03.461+0000 7fb29f7fe700 1 --2- 172.21.15.194:0/1263575989 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb2a002ff60 0x7fb2a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:03.463 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:03.461+0000 7fb29f7fe700 1 -- 172.21.15.194:0/1263575989 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb2ac041000 con 0x7fb2b80f5410 2024-01-18T06:53:03.463 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:03.461+0000 7fb2b67fc700 1 --2- 172.21.15.194:0/1263575989 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb2a002ff60 0x7fb2a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:03.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:03 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:53:02.016300+0000 mgr.smithi047.fuhzzr (mgr.14162) 137 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:53:03.463 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:03.462+0000 7fb2b67fc700 1 --2- 172.21.15.194:0/1263575989 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb2a002ff60 0x7fb2a0032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fb2a80096a0 tx=0x7fb2a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:03.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:03.466+0000 7fb29f7fe700 1 -- 172.21.15.194:0/1263575989 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb2ac0095f0 con 0x7fb2b80f5410 2024-01-18T06:53:03.672 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:03.672+0000 7fb2bd520700 1 -- 172.21.15.194:0/1263575989 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb2b80fae60 con 0x7fb2b80f5410 2024-01-18T06:53:03.673 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:03.672+0000 7fb29f7fe700 1 -- 172.21.15.194:0/1263575989 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb2ac01db70 con 0x7fb2b80f5410 2024-01-18T06:53:03.673 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:53:03.673 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:53:03.674 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:03.674+0000 7fb2bd520700 1 -- 172.21.15.194:0/1263575989 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb2a002ff60 msgr2=0x7fb2a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:03.675 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:03.674+0000 7fb2bd520700 1 --2- 172.21.15.194:0/1263575989 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb2a002ff60 0x7fb2a0032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fb2a80096a0 tx=0x7fb2a8006b40).stop 2024-01-18T06:53:03.675 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:03.674+0000 7fb2bd520700 1 -- 172.21.15.194:0/1263575989 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb2b80f5410 msgr2=0x7fb2b80fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:03.675 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:03.674+0000 7fb2bd520700 1 --2- 172.21.15.194:0/1263575989 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb2b80f5410 0x7fb2b80fd3a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7fb2ac013510 tx=0x7fb2ac00b040).stop 2024-01-18T06:53:03.675 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:03.674+0000 7fb2bd520700 1 -- 172.21.15.194:0/1263575989 shutdown_connections 2024-01-18T06:53:03.675 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:03.674+0000 7fb2bd520700 1 --2- 172.21.15.194:0/1263575989 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb2a002ff60 0x7fb2a0032420 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:03.675 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:03.674+0000 7fb2bd520700 1 --2- 172.21.15.194:0/1263575989 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb2b80f5410 0x7fb2b80fd3a0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:03.675 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:03.674+0000 7fb2bd520700 1 -- 172.21.15.194:0/1263575989 >> 172.21.15.194:0/1263575989 conn(0x7fb2b80f0b70 msgr2=0x7fb2b805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:03.675 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:03.674+0000 7fb2bd520700 1 -- 172.21.15.194:0/1263575989 shutdown_connections 2024-01-18T06:53:03.675 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:03.674+0000 7fb2bd520700 1 -- 172.21.15.194:0/1263575989 wait complete. 2024-01-18T06:53:03.675 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:53:04.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:04 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:53:03.674477+0000 mon.smithi047 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.194:0/1263575989' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:53:05.082 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:53:05.082 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:53:05.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:05 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:53:04.016676+0000 mgr.smithi047.fuhzzr (mgr.14162) 138 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:53:06.281 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:06.278+0000 7fb1eb1c1700 1 -- 172.21.15.194:0/1377683565 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1e40f5410 msgr2=0x7fb1e40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:06.281 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:06.278+0000 7fb1eb1c1700 1 --2- 172.21.15.194:0/1377683565 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1e40f5410 0x7fb1e40f5830 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7fb1d8009fb0 tx=0x7fb1d8009b30).stop 2024-01-18T06:53:06.281 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:06.278+0000 7fb1eb1c1700 1 -- 172.21.15.194:0/1377683565 shutdown_connections 2024-01-18T06:53:06.281 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:06.278+0000 7fb1eb1c1700 1 --2- 172.21.15.194:0/1377683565 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1e40f5410 0x7fb1e40f5830 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:06.281 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:06.278+0000 7fb1eb1c1700 1 -- 172.21.15.194:0/1377683565 >> 172.21.15.194:0/1377683565 conn(0x7fb1e40f0b70 msgr2=0x7fb1e40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:06.282 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:06.278+0000 7fb1eb1c1700 1 -- 172.21.15.194:0/1377683565 shutdown_connections 2024-01-18T06:53:06.282 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:06.278+0000 7fb1eb1c1700 1 -- 172.21.15.194:0/1377683565 wait complete. 2024-01-18T06:53:06.282 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:06.279+0000 7fb1eb1c1700 1 Processor -- start 2024-01-18T06:53:06.282 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:06.279+0000 7fb1eb1c1700 1 -- start start 2024-01-18T06:53:06.282 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:06.279+0000 7fb1eb1c1700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1e40f5410 0x7fb1e4109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:06.282 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:06.279+0000 7fb1eb1c1700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e4060220 con 0x7fb1e40f5410 2024-01-18T06:53:06.282 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:06.280+0000 7fb1e8f5d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1e40f5410 0x7fb1e4109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:06.282 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:06.280+0000 7fb1e8f5d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1e40f5410 0x7fb1e4109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:40996/0 (socket says 172.21.15.194:40996) 2024-01-18T06:53:06.282 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:06.280+0000 7fb1e8f5d700 1 -- 172.21.15.194:0/1624775918 learned_addr learned my addr 172.21.15.194:0/1624775918 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:53:06.282 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:06.280+0000 7fb1e8f5d700 1 -- 172.21.15.194:0/1624775918 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d8005040 con 0x7fb1e40f5410 2024-01-18T06:53:06.282 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:06.280+0000 7fb1e8f5d700 1 --2- 172.21.15.194:0/1624775918 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1e40f5410 0x7fb1e4109450 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fb1d8013510 tx=0x7fb1d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:06.282 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:06.280+0000 7fb1e17fa700 1 -- 172.21.15.194:0/1624775918 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1d8009320 con 0x7fb1e40f5410 2024-01-18T06:53:06.282 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:06.280+0000 7fb1eb1c1700 1 -- 172.21.15.194:0/1624775918 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1e4060420 con 0x7fb1e40f5410 2024-01-18T06:53:06.283 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:06.280+0000 7fb1e17fa700 1 -- 172.21.15.194:0/1624775918 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb1d8013f00 con 0x7fb1e40f5410 2024-01-18T06:53:06.283 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:06.280+0000 7fb1e17fa700 1 -- 172.21.15.194:0/1624775918 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1d8014d90 con 0x7fb1e40f5410 2024-01-18T06:53:06.283 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:06.280+0000 7fb1eb1c1700 1 -- 172.21.15.194:0/1624775918 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1e4109b00 con 0x7fb1e40f5410 2024-01-18T06:53:06.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:06.281+0000 7fb1e17fa700 1 -- 172.21.15.194:0/1624775918 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb1d80263a0 con 0x7fb1e40f5410 2024-01-18T06:53:06.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:06.281+0000 7fb1e17fa700 1 --2- 172.21.15.194:0/1624775918 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb1d402ff60 0x7fb1d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:06.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:06.281+0000 7fb1e17fa700 1 -- 172.21.15.194:0/1624775918 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb1d8041090 con 0x7fb1e40f5410 2024-01-18T06:53:06.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:06.281+0000 7fb1eb1c1700 1 -- 172.21.15.194:0/1624775918 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1e4059100 con 0x7fb1e40f5410 2024-01-18T06:53:06.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:06.282+0000 7fb1e3fff700 1 --2- 172.21.15.194:0/1624775918 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb1d402ff60 0x7fb1d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:06.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:06.282+0000 7fb1e3fff700 1 --2- 172.21.15.194:0/1624775918 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb1d402ff60 0x7fb1d4032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fb1d00096a0 tx=0x7fb1d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:06.287 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:06.286+0000 7fb1e17fa700 1 -- 172.21.15.194:0/1624775918 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb1d8012920 con 0x7fb1e40f5410 2024-01-18T06:53:06.490 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:06.490+0000 7fb1eb1c1700 1 -- 172.21.15.194:0/1624775918 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb1e4045ad0 con 0x7fb1e40f5410 2024-01-18T06:53:06.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:06.491+0000 7fb1e17fa700 1 -- 172.21.15.194:0/1624775918 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb1d802b020 con 0x7fb1e40f5410 2024-01-18T06:53:06.491 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:53:06.491 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:53:06.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:06.492+0000 7fb1eb1c1700 1 -- 172.21.15.194:0/1624775918 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb1d402ff60 msgr2=0x7fb1d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:06.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:06.492+0000 7fb1eb1c1700 1 --2- 172.21.15.194:0/1624775918 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb1d402ff60 0x7fb1d4032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fb1d00096a0 tx=0x7fb1d0006b40).stop 2024-01-18T06:53:06.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:06.492+0000 7fb1eb1c1700 1 -- 172.21.15.194:0/1624775918 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1e40f5410 msgr2=0x7fb1e4109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:06.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:06.492+0000 7fb1eb1c1700 1 --2- 172.21.15.194:0/1624775918 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1e40f5410 0x7fb1e4109450 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fb1d8013510 tx=0x7fb1d800b040).stop 2024-01-18T06:53:06.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:06.492+0000 7fb1eb1c1700 1 -- 172.21.15.194:0/1624775918 shutdown_connections 2024-01-18T06:53:06.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:06.492+0000 7fb1eb1c1700 1 --2- 172.21.15.194:0/1624775918 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb1d402ff60 0x7fb1d4032420 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:06.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:06.492+0000 7fb1eb1c1700 1 --2- 172.21.15.194:0/1624775918 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1e40f5410 0x7fb1e4109450 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:06.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:06.492+0000 7fb1eb1c1700 1 -- 172.21.15.194:0/1624775918 >> 172.21.15.194:0/1624775918 conn(0x7fb1e40f0b70 msgr2=0x7fb1e40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:06.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:06.492+0000 7fb1eb1c1700 1 -- 172.21.15.194:0/1624775918 shutdown_connections 2024-01-18T06:53:06.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:06.492+0000 7fb1eb1c1700 1 -- 172.21.15.194:0/1624775918 wait complete. 2024-01-18T06:53:06.494 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:53:07.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:07 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:53:06.017029+0000 mgr.smithi047.fuhzzr (mgr.14162) 139 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:53:07.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:07 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:53:06.492645+0000 mon.smithi047 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.194:0/1624775918' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:53:07.875 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:53:07.875 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:53:09.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:09.060+0000 7f5c5b59e700 1 -- 172.21.15.194:0/3201934259 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c5c0f51a0 msgr2=0x7f5c5c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:09.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:09.060+0000 7f5c5b59e700 1 --2- 172.21.15.194:0/3201934259 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c5c0f51a0 0x7f5c5c0f55c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f5c50009fb0 tx=0x7f5c50009b30).stop 2024-01-18T06:53:09.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:09.060+0000 7f5c5b59e700 1 -- 172.21.15.194:0/3201934259 shutdown_connections 2024-01-18T06:53:09.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:09.060+0000 7f5c5b59e700 1 --2- 172.21.15.194:0/3201934259 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c5c0f51a0 0x7f5c5c0f55c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:09.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:09.060+0000 7f5c5b59e700 1 -- 172.21.15.194:0/3201934259 >> 172.21.15.194:0/3201934259 conn(0x7f5c5c0f09a0 msgr2=0x7f5c5c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:09.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:09.060+0000 7f5c5b59e700 1 -- 172.21.15.194:0/3201934259 shutdown_connections 2024-01-18T06:53:09.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:09.060+0000 7f5c5b59e700 1 -- 172.21.15.194:0/3201934259 wait complete. 2024-01-18T06:53:09.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:09.060+0000 7f5c5b59e700 1 Processor -- start 2024-01-18T06:53:09.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:09.061+0000 7f5c5b59e700 1 -- start start 2024-01-18T06:53:09.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:09.061+0000 7f5c5b59e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c5c0f51a0 0x7f5c5c1053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:09.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:09.061+0000 7f5c5b59e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5c1058a0 con 0x7f5c5c0f51a0 2024-01-18T06:53:09.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:09.061+0000 7f5c5a59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c5c0f51a0 0x7f5c5c1053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:09.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:09.061+0000 7f5c5a59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c5c0f51a0 0x7f5c5c1053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:38474/0 (socket says 172.21.15.194:38474) 2024-01-18T06:53:09.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:09.061+0000 7f5c5a59c700 1 -- 172.21.15.194:0/1805036163 learned_addr learned my addr 172.21.15.194:0/1805036163 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:53:09.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:09.062+0000 7f5c5a59c700 1 -- 172.21.15.194:0/1805036163 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c50005040 con 0x7f5c5c0f51a0 2024-01-18T06:53:09.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:09.062+0000 7f5c5a59c700 1 --2- 172.21.15.194:0/1805036163 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c5c0f51a0 0x7f5c5c1053b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f5c50013510 tx=0x7f5c5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:09.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:09.062+0000 7f5c4affd700 1 -- 172.21.15.194:0/1805036163 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c50009320 con 0x7f5c5c0f51a0 2024-01-18T06:53:09.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:09.062+0000 7f5c5b59e700 1 -- 172.21.15.194:0/1805036163 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c5c105aa0 con 0x7f5c5c0f51a0 2024-01-18T06:53:09.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:09.062+0000 7f5c4affd700 1 -- 172.21.15.194:0/1805036163 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5c50013f00 con 0x7f5c5c0f51a0 2024-01-18T06:53:09.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:09.062+0000 7f5c4affd700 1 -- 172.21.15.194:0/1805036163 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c50014dc0 con 0x7f5c5c0f51a0 2024-01-18T06:53:09.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:09.062+0000 7f5c5b59e700 1 -- 172.21.15.194:0/1805036163 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c5c105f30 con 0x7f5c5c0f51a0 2024-01-18T06:53:09.064 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:09.063+0000 7f5c4affd700 1 -- 172.21.15.194:0/1805036163 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5c5001d3d0 con 0x7f5c5c0f51a0 2024-01-18T06:53:09.064 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:09.063+0000 7f5c4affd700 1 --2- 172.21.15.194:0/1805036163 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f5c4402ff60 0x7f5c44032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:09.065 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:09.063+0000 7f5c4affd700 1 -- 172.21.15.194:0/1805036163 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5c50041120 con 0x7f5c5c0f51a0 2024-01-18T06:53:09.065 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:09.063+0000 7f5c5b59e700 1 -- 172.21.15.194:0/1805036163 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c5c045ad0 con 0x7f5c5c0f51a0 2024-01-18T06:53:09.065 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:09.063+0000 7f5c59d9b700 1 --2- 172.21.15.194:0/1805036163 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f5c4402ff60 0x7f5c44032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:09.065 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:09.064+0000 7f5c59d9b700 1 --2- 172.21.15.194:0/1805036163 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f5c4402ff60 0x7f5c44032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f5c4c0096a0 tx=0x7f5c4c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:09.068 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:09.069+0000 7f5c4affd700 1 -- 172.21.15.194:0/1805036163 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5c50012920 con 0x7f5c5c0f51a0 2024-01-18T06:53:09.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:08 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:53:08.017362+0000 mgr.smithi047.fuhzzr (mgr.14162) 140 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:53:09.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:09.272+0000 7f5c5b59e700 1 -- 172.21.15.194:0/1805036163 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5c5c106c50 con 0x7f5c5c0f51a0 2024-01-18T06:53:09.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:09.274+0000 7f5c4affd700 1 -- 172.21.15.194:0/1805036163 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5c5002b020 con 0x7f5c5c0f51a0 2024-01-18T06:53:09.274 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:53:09.274 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:53:09.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:09.275+0000 7f5c5b59e700 1 -- 172.21.15.194:0/1805036163 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f5c4402ff60 msgr2=0x7f5c44032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:09.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:09.275+0000 7f5c5b59e700 1 --2- 172.21.15.194:0/1805036163 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f5c4402ff60 0x7f5c44032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f5c4c0096a0 tx=0x7f5c4c006b40).stop 2024-01-18T06:53:09.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:09.275+0000 7f5c5b59e700 1 -- 172.21.15.194:0/1805036163 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c5c0f51a0 msgr2=0x7f5c5c1053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:09.276 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:09.275+0000 7f5c5b59e700 1 --2- 172.21.15.194:0/1805036163 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c5c0f51a0 0x7f5c5c1053b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f5c50013510 tx=0x7f5c5000b040).stop 2024-01-18T06:53:09.276 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:09.275+0000 7f5c5b59e700 1 -- 172.21.15.194:0/1805036163 shutdown_connections 2024-01-18T06:53:09.276 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:09.275+0000 7f5c5b59e700 1 --2- 172.21.15.194:0/1805036163 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f5c4402ff60 0x7f5c44032420 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:09.276 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:09.275+0000 7f5c5b59e700 1 --2- 172.21.15.194:0/1805036163 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c5c0f51a0 0x7f5c5c1053b0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:09.276 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:09.275+0000 7f5c5b59e700 1 -- 172.21.15.194:0/1805036163 >> 172.21.15.194:0/1805036163 conn(0x7f5c5c0f09a0 msgr2=0x7f5c5c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:09.276 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:09.275+0000 7f5c5b59e700 1 -- 172.21.15.194:0/1805036163 shutdown_connections 2024-01-18T06:53:09.276 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:09.275+0000 7f5c5b59e700 1 -- 172.21.15.194:0/1805036163 wait complete. 2024-01-18T06:53:09.276 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:53:10.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:09 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:53:09.274923+0000 mon.smithi047 (mon.0) 213 : audit [DBG] from='client.? 172.21.15.194:0/1805036163' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:53:10.666 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:53:10.667 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:53:11.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:10 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:53:10.017682+0000 mgr.smithi047.fuhzzr (mgr.14162) 141 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:53:11.818 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:11.817+0000 7f3ec759e700 1 -- 172.21.15.194:0/2736230477 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3ec80f51a0 msgr2=0x7f3ec80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:11.818 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:11.817+0000 7f3ec759e700 1 --2- 172.21.15.194:0/2736230477 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3ec80f51a0 0x7f3ec80f55c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f3eb8009fb0 tx=0x7f3eb8009b30).stop 2024-01-18T06:53:11.818 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:11.817+0000 7f3ec759e700 1 -- 172.21.15.194:0/2736230477 shutdown_connections 2024-01-18T06:53:11.818 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:11.817+0000 7f3ec759e700 1 --2- 172.21.15.194:0/2736230477 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3ec80f51a0 0x7f3ec80f55c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:11.818 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:11.817+0000 7f3ec759e700 1 -- 172.21.15.194:0/2736230477 >> 172.21.15.194:0/2736230477 conn(0x7f3ec80f09a0 msgr2=0x7f3ec80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:11.818 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:11.817+0000 7f3ec759e700 1 -- 172.21.15.194:0/2736230477 shutdown_connections 2024-01-18T06:53:11.818 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:11.817+0000 7f3ec759e700 1 -- 172.21.15.194:0/2736230477 wait complete. 2024-01-18T06:53:11.818 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:11.817+0000 7f3ec759e700 1 Processor -- start 2024-01-18T06:53:11.819 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:11.817+0000 7f3ec759e700 1 -- start start 2024-01-18T06:53:11.819 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:11.817+0000 7f3ec759e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3ec80f51a0 0x7f3ec8109200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:11.819 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:11.817+0000 7f3ec759e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec8060220 con 0x7f3ec80f51a0 2024-01-18T06:53:11.819 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:11.818+0000 7f3ec659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3ec80f51a0 0x7f3ec8109200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:11.819 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:11.818+0000 7f3ec659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3ec80f51a0 0x7f3ec8109200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:38494/0 (socket says 172.21.15.194:38494) 2024-01-18T06:53:11.819 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:11.818+0000 7f3ec659c700 1 -- 172.21.15.194:0/3425193348 learned_addr learned my addr 172.21.15.194:0/3425193348 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:53:11.819 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:11.818+0000 7f3ec659c700 1 -- 172.21.15.194:0/3425193348 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb8005040 con 0x7f3ec80f51a0 2024-01-18T06:53:11.819 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:11.818+0000 7f3ec659c700 1 --2- 172.21.15.194:0/3425193348 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3ec80f51a0 0x7f3ec8109200 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f3eb8013510 tx=0x7f3eb800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:11.819 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:11.818+0000 7f3eb6ffd700 1 -- 172.21.15.194:0/3425193348 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3eb8009320 con 0x7f3ec80f51a0 2024-01-18T06:53:11.819 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:11.818+0000 7f3ec759e700 1 -- 172.21.15.194:0/3425193348 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ec8060420 con 0x7f3ec80f51a0 2024-01-18T06:53:11.819 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:11.818+0000 7f3eb6ffd700 1 -- 172.21.15.194:0/3425193348 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3eb8013f00 con 0x7f3ec80f51a0 2024-01-18T06:53:11.819 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:11.818+0000 7f3eb6ffd700 1 -- 172.21.15.194:0/3425193348 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3eb8014dc0 con 0x7f3ec80f51a0 2024-01-18T06:53:11.819 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:11.818+0000 7f3ec759e700 1 -- 172.21.15.194:0/3425193348 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ec81098b0 con 0x7f3ec80f51a0 2024-01-18T06:53:11.820 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:11.819+0000 7f3eb6ffd700 1 -- 172.21.15.194:0/3425193348 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3eb80095f0 con 0x7f3ec80f51a0 2024-01-18T06:53:11.820 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:11.819+0000 7f3ec759e700 1 -- 172.21.15.194:0/3425193348 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ec8059100 con 0x7f3ec80f51a0 2024-01-18T06:53:11.820 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:11.819+0000 7f3eb6ffd700 1 --2- 172.21.15.194:0/3425193348 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3eb002ff60 0x7f3eb0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:11.821 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:11.819+0000 7f3eb6ffd700 1 -- 172.21.15.194:0/3425193348 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3eb8041020 con 0x7f3ec80f51a0 2024-01-18T06:53:11.821 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:11.820+0000 7f3ec5d9b700 1 --2- 172.21.15.194:0/3425193348 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3eb002ff60 0x7f3eb0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:11.821 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:11.820+0000 7f3ec5d9b700 1 --2- 172.21.15.194:0/3425193348 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3eb002ff60 0x7f3eb0032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f3ebc0096a0 tx=0x7f3ebc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:11.823 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:11.824+0000 7f3eb6ffd700 1 -- 172.21.15.194:0/3425193348 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3eb8014420 con 0x7f3ec80f51a0 2024-01-18T06:53:12.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:12.029+0000 7f3ec759e700 1 -- 172.21.15.194:0/3425193348 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3ec80fabe0 con 0x7f3ec80f51a0 2024-01-18T06:53:12.029 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:12.030+0000 7f3eb6ffd700 1 -- 172.21.15.194:0/3425193348 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3eb801c920 con 0x7f3ec80f51a0 2024-01-18T06:53:12.029 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:53:12.029 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:53:12.031 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:12.031+0000 7f3ec759e700 1 -- 172.21.15.194:0/3425193348 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3eb002ff60 msgr2=0x7f3eb0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:12.031 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:12.031+0000 7f3ec759e700 1 --2- 172.21.15.194:0/3425193348 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3eb002ff60 0x7f3eb0032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f3ebc0096a0 tx=0x7f3ebc006b40).stop 2024-01-18T06:53:12.031 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:12.031+0000 7f3ec759e700 1 -- 172.21.15.194:0/3425193348 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3ec80f51a0 msgr2=0x7f3ec8109200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:12.031 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:12.031+0000 7f3ec759e700 1 --2- 172.21.15.194:0/3425193348 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3ec80f51a0 0x7f3ec8109200 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f3eb8013510 tx=0x7f3eb800b040).stop 2024-01-18T06:53:12.031 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:12.031+0000 7f3ec759e700 1 -- 172.21.15.194:0/3425193348 shutdown_connections 2024-01-18T06:53:12.031 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:12.032+0000 7f3ec759e700 1 --2- 172.21.15.194:0/3425193348 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3eb002ff60 0x7f3eb0032420 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:12.031 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:12.032+0000 7f3ec759e700 1 --2- 172.21.15.194:0/3425193348 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3ec80f51a0 0x7f3ec8109200 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:12.032 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:12.032+0000 7f3ec759e700 1 -- 172.21.15.194:0/3425193348 >> 172.21.15.194:0/3425193348 conn(0x7f3ec80f09a0 msgr2=0x7f3ec80f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:12.032 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:12.032+0000 7f3ec759e700 1 -- 172.21.15.194:0/3425193348 shutdown_connections 2024-01-18T06:53:12.032 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:12.032+0000 7f3ec759e700 1 -- 172.21.15.194:0/3425193348 wait complete. 2024-01-18T06:53:12.032 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:53:12.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:12 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:53:12.030563+0000 mon.smithi047 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.194:0/3425193348' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:53:13.415 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:53:13.416 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:53:13.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:13 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:53:12.017981+0000 mgr.smithi047.fuhzzr (mgr.14162) 142 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:53:14.686 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:14.685+0000 7efef0620700 1 -- 172.21.15.194:0/4229104820 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efee80f7b60 msgr2=0x7efee80f7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:14.686 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:14.685+0000 7efef0620700 1 --2- 172.21.15.194:0/4229104820 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efee80f7b60 0x7efee80f7f80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7efed8009fb0 tx=0x7efed8009b30).stop 2024-01-18T06:53:14.687 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:14.685+0000 7efef0620700 1 -- 172.21.15.194:0/4229104820 shutdown_connections 2024-01-18T06:53:14.687 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:14.685+0000 7efef0620700 1 --2- 172.21.15.194:0/4229104820 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efee80f7b60 0x7efee80f7f80 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:14.687 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:14.685+0000 7efef0620700 1 -- 172.21.15.194:0/4229104820 >> 172.21.15.194:0/4229104820 conn(0x7efee80f02e0 msgr2=0x7efee80f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:14.687 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:14.685+0000 7efef0620700 1 -- 172.21.15.194:0/4229104820 shutdown_connections 2024-01-18T06:53:14.687 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:14.685+0000 7efef0620700 1 -- 172.21.15.194:0/4229104820 wait complete. 2024-01-18T06:53:14.687 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:14.686+0000 7efef0620700 1 Processor -- start 2024-01-18T06:53:14.687 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:14.686+0000 7efef0620700 1 -- start start 2024-01-18T06:53:14.687 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:14.686+0000 7efef0620700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efee80f7b60 0x7efee8106740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:14.687 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:14.686+0000 7efef0620700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee8106c30 con 0x7efee80f7b60 2024-01-18T06:53:14.687 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:14.686+0000 7efeee3bc700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efee80f7b60 0x7efee8106740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:14.687 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:14.686+0000 7efeee3bc700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efee80f7b60 0x7efee8106740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:38516/0 (socket says 172.21.15.194:38516) 2024-01-18T06:53:14.687 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:14.686+0000 7efeee3bc700 1 -- 172.21.15.194:0/2341286992 learned_addr learned my addr 172.21.15.194:0/2341286992 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:53:14.687 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:14.687+0000 7efeee3bc700 1 -- 172.21.15.194:0/2341286992 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed8005040 con 0x7efee80f7b60 2024-01-18T06:53:14.688 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:14.687+0000 7efeee3bc700 1 --2- 172.21.15.194:0/2341286992 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efee80f7b60 0x7efee8106740 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7efed8013510 tx=0x7efed800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:14.688 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:14.687+0000 7efedeffd700 1 -- 172.21.15.194:0/2341286992 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efed8009320 con 0x7efee80f7b60 2024-01-18T06:53:14.688 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:14.687+0000 7efef0620700 1 -- 172.21.15.194:0/2341286992 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efee8106e30 con 0x7efee80f7b60 2024-01-18T06:53:14.688 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:14.687+0000 7efedeffd700 1 -- 172.21.15.194:0/2341286992 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efed8013f00 con 0x7efee80f7b60 2024-01-18T06:53:14.688 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:14.687+0000 7efedeffd700 1 -- 172.21.15.194:0/2341286992 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efed8014d90 con 0x7efee80f7b60 2024-01-18T06:53:14.688 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:14.687+0000 7efef0620700 1 -- 172.21.15.194:0/2341286992 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efee81072c0 con 0x7efee80f7b60 2024-01-18T06:53:14.689 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:14.688+0000 7efedeffd700 1 -- 172.21.15.194:0/2341286992 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efed8014420 con 0x7efee80f7b60 2024-01-18T06:53:14.689 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:14.688+0000 7efef0620700 1 -- 172.21.15.194:0/2341286992 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efee8045ad0 con 0x7efee80f7b60 2024-01-18T06:53:14.689 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:14.688+0000 7efedeffd700 1 --2- 172.21.15.194:0/2341286992 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7efed402ff60 0x7efed4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:14.689 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:14.688+0000 7efedeffd700 1 -- 172.21.15.194:0/2341286992 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efed8041000 con 0x7efee80f7b60 2024-01-18T06:53:14.689 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:14.688+0000 7efeedbbb700 1 --2- 172.21.15.194:0/2341286992 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7efed402ff60 0x7efed4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:14.689 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:14.688+0000 7efeedbbb700 1 --2- 172.21.15.194:0/2341286992 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7efed402ff60 0x7efed4032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7efee40096a0 tx=0x7efee4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:14.692 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:14.693+0000 7efedeffd700 1 -- 172.21.15.194:0/2341286992 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efed80095f0 con 0x7efee80f7b60 2024-01-18T06:53:14.901 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:14.901+0000 7efef0620700 1 -- 172.21.15.194:0/2341286992 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efee8108170 con 0x7efee80f7b60 2024-01-18T06:53:14.901 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:14.902+0000 7efedeffd700 1 -- 172.21.15.194:0/2341286992 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efed801db70 con 0x7efee80f7b60 2024-01-18T06:53:14.901 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:53:14.902 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:53:14.903 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:14.903+0000 7efef0620700 1 -- 172.21.15.194:0/2341286992 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7efed402ff60 msgr2=0x7efed4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:14.903 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:14.903+0000 7efef0620700 1 --2- 172.21.15.194:0/2341286992 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7efed402ff60 0x7efed4032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7efee40096a0 tx=0x7efee4006b40).stop 2024-01-18T06:53:14.903 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:14.903+0000 7efef0620700 1 -- 172.21.15.194:0/2341286992 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efee80f7b60 msgr2=0x7efee8106740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:14.903 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:14.903+0000 7efef0620700 1 --2- 172.21.15.194:0/2341286992 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efee80f7b60 0x7efee8106740 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7efed8013510 tx=0x7efed800b040).stop 2024-01-18T06:53:14.903 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:14.903+0000 7efef0620700 1 -- 172.21.15.194:0/2341286992 shutdown_connections 2024-01-18T06:53:14.903 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:14.903+0000 7efef0620700 1 --2- 172.21.15.194:0/2341286992 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7efed402ff60 0x7efed4032420 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:14.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:14.903+0000 7efef0620700 1 --2- 172.21.15.194:0/2341286992 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efee80f7b60 0x7efee8106740 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:14.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:14.903+0000 7efef0620700 1 -- 172.21.15.194:0/2341286992 >> 172.21.15.194:0/2341286992 conn(0x7efee80f02e0 msgr2=0x7efee80f0e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:14.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:14.904+0000 7efef0620700 1 -- 172.21.15.194:0/2341286992 shutdown_connections 2024-01-18T06:53:14.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:14.904+0000 7efef0620700 1 -- 172.21.15.194:0/2341286992 wait complete. 2024-01-18T06:53:14.904 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:53:15.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:15 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:53:14.018299+0000 mgr.smithi047.fuhzzr (mgr.14162) 143 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:53:15.464 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:15 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:53:14.902764+0000 mon.smithi047 (mon.0) 215 : audit [DBG] from='client.? 172.21.15.194:0/2341286992' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:53:16.301 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:53:16.301 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:53:17.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:17 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:53:16.018660+0000 mgr.smithi047.fuhzzr (mgr.14162) 144 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:53:17.465 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:17.465+0000 7f60bd317700 1 -- 172.21.15.194:0/2277673762 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60b80f62a0 msgr2=0x7f60b80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:17.465 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:17.465+0000 7f60bd317700 1 --2- 172.21.15.194:0/2277673762 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60b80f62a0 0x7f60b80f66c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f60a8009fb0 tx=0x7f60a8009b30).stop 2024-01-18T06:53:17.465 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:17.465+0000 7f60bd317700 1 -- 172.21.15.194:0/2277673762 shutdown_connections 2024-01-18T06:53:17.465 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:17.465+0000 7f60bd317700 1 --2- 172.21.15.194:0/2277673762 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60b80f62a0 0x7f60b80f66c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:17.465 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:17.465+0000 7f60bd317700 1 -- 172.21.15.194:0/2277673762 >> 172.21.15.194:0/2277673762 conn(0x7f60b80f1aa0 msgr2=0x7f60b80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:17.465 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:17.465+0000 7f60bd317700 1 -- 172.21.15.194:0/2277673762 shutdown_connections 2024-01-18T06:53:17.465 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:17.465+0000 7f60bd317700 1 -- 172.21.15.194:0/2277673762 wait complete. 2024-01-18T06:53:17.465 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:17.465+0000 7f60bd317700 1 Processor -- start 2024-01-18T06:53:17.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:17.465+0000 7f60bd317700 1 -- start start 2024-01-18T06:53:17.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:17.466+0000 7f60bd317700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60b80f62a0 0x7f60b81064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:17.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:17.466+0000 7f60bd317700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b81069a0 con 0x7f60b80f62a0 2024-01-18T06:53:17.467 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:17.466+0000 7f60b7fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60b80f62a0 0x7f60b81064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:17.468 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:17.466+0000 7f60b7fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60b80f62a0 0x7f60b81064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:48146/0 (socket says 172.21.15.194:48146) 2024-01-18T06:53:17.468 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:17.466+0000 7f60b7fff700 1 -- 172.21.15.194:0/3225404852 learned_addr learned my addr 172.21.15.194:0/3225404852 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:53:17.468 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:17.466+0000 7f60b7fff700 1 -- 172.21.15.194:0/3225404852 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60a8005040 con 0x7f60b80f62a0 2024-01-18T06:53:17.468 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:17.466+0000 7f60b7fff700 1 --2- 172.21.15.194:0/3225404852 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60b80f62a0 0x7f60b81064b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f60a8009830 tx=0x7f60a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:17.468 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:17.467+0000 7f60b4ff9700 1 -- 172.21.15.194:0/3225404852 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60a8009320 con 0x7f60b80f62a0 2024-01-18T06:53:17.468 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:17.467+0000 7f60b4ff9700 1 -- 172.21.15.194:0/3225404852 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f60a80136c0 con 0x7f60b80f62a0 2024-01-18T06:53:17.468 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:17.467+0000 7f60bd317700 1 -- 172.21.15.194:0/3225404852 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60b8106ba0 con 0x7f60b80f62a0 2024-01-18T06:53:17.468 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:17.467+0000 7f60b4ff9700 1 -- 172.21.15.194:0/3225404852 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60a801ee90 con 0x7f60b80f62a0 2024-01-18T06:53:17.468 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:17.467+0000 7f60bd317700 1 -- 172.21.15.194:0/3225404852 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60b8107030 con 0x7f60b80f62a0 2024-01-18T06:53:17.468 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:17.467+0000 7f60b4ff9700 1 -- 172.21.15.194:0/3225404852 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f60a801e420 con 0x7f60b80f62a0 2024-01-18T06:53:17.468 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:17.467+0000 7f60bd317700 1 -- 172.21.15.194:0/3225404852 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60b8045ad0 con 0x7f60b80f62a0 2024-01-18T06:53:17.471 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:17.468+0000 7f60b4ff9700 1 --2- 172.21.15.194:0/3225404852 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f60a002ff10 0x7f60a00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:17.471 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:17.468+0000 7f60b4ff9700 1 -- 172.21.15.194:0/3225404852 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f60a8040f40 con 0x7f60b80f62a0 2024-01-18T06:53:17.471 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:17.468+0000 7f60b77fe700 1 --2- 172.21.15.194:0/3225404852 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f60a002ff10 0x7f60a00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:17.471 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:17.468+0000 7f60b77fe700 1 --2- 172.21.15.194:0/3225404852 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f60a002ff10 0x7f60a00323d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f60ac0096a0 tx=0x7f60ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:17.472 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:17.472+0000 7f60b4ff9700 1 -- 172.21.15.194:0/3225404852 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f60a806d2b0 con 0x7f60b80f62a0 2024-01-18T06:53:17.677 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:17.678+0000 7f60bd317700 1 -- 172.21.15.194:0/3225404852 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f60b8107ee0 con 0x7f60b80f62a0 2024-01-18T06:53:17.678 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:17.679+0000 7f60b4ff9700 1 -- 172.21.15.194:0/3225404852 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f60a80095f0 con 0x7f60b80f62a0 2024-01-18T06:53:17.678 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:53:17.678 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:53:17.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:17.680+0000 7f60bd317700 1 -- 172.21.15.194:0/3225404852 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f60a002ff10 msgr2=0x7f60a00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:17.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:17.680+0000 7f60bd317700 1 --2- 172.21.15.194:0/3225404852 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f60a002ff10 0x7f60a00323d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f60ac0096a0 tx=0x7f60ac006b40).stop 2024-01-18T06:53:17.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:17.680+0000 7f60bd317700 1 -- 172.21.15.194:0/3225404852 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60b80f62a0 msgr2=0x7f60b81064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:17.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:17.680+0000 7f60bd317700 1 --2- 172.21.15.194:0/3225404852 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60b80f62a0 0x7f60b81064b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f60a8009830 tx=0x7f60a800b040).stop 2024-01-18T06:53:17.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:17.680+0000 7f60bd317700 1 -- 172.21.15.194:0/3225404852 shutdown_connections 2024-01-18T06:53:17.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:17.680+0000 7f60bd317700 1 --2- 172.21.15.194:0/3225404852 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f60a002ff10 0x7f60a00323d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:17.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:17.680+0000 7f60bd317700 1 --2- 172.21.15.194:0/3225404852 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60b80f62a0 0x7f60b81064b0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:17.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:17.680+0000 7f60bd317700 1 -- 172.21.15.194:0/3225404852 >> 172.21.15.194:0/3225404852 conn(0x7f60b80f1aa0 msgr2=0x7f60b80f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:17.681 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:17.680+0000 7f60bd317700 1 -- 172.21.15.194:0/3225404852 shutdown_connections 2024-01-18T06:53:17.681 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:17.680+0000 7f60bd317700 1 -- 172.21.15.194:0/3225404852 wait complete. 2024-01-18T06:53:17.681 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:53:18.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:18 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:53:17.679734+0000 mon.smithi047 (mon.0) 216 : audit [DBG] from='client.? 172.21.15.194:0/3225404852' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:53:19.060 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:53:19.060 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:53:19.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:19 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:53:18.018906+0000 mgr.smithi047.fuhzzr (mgr.14162) 145 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:53:20.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:20.283+0000 7f45ab59e700 1 -- 172.21.15.194:0/3363758383 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45ac0f78e0 msgr2=0x7f45ac0f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:20.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:20.283+0000 7f45ab59e700 1 --2- 172.21.15.194:0/3363758383 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45ac0f78e0 0x7f45ac0f7d00 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f459c009fb0 tx=0x7f459c009b30).stop 2024-01-18T06:53:20.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:20.283+0000 7f45ab59e700 1 -- 172.21.15.194:0/3363758383 shutdown_connections 2024-01-18T06:53:20.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:20.283+0000 7f45ab59e700 1 --2- 172.21.15.194:0/3363758383 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45ac0f78e0 0x7f45ac0f7d00 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:20.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:20.283+0000 7f45ab59e700 1 -- 172.21.15.194:0/3363758383 >> 172.21.15.194:0/3363758383 conn(0x7f45ac0f0120 msgr2=0x7f45ac0f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:20.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:20.283+0000 7f45ab59e700 1 -- 172.21.15.194:0/3363758383 shutdown_connections 2024-01-18T06:53:20.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:20.283+0000 7f45ab59e700 1 -- 172.21.15.194:0/3363758383 wait complete. 2024-01-18T06:53:20.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:20.283+0000 7f45ab59e700 1 Processor -- start 2024-01-18T06:53:20.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:20.283+0000 7f45ab59e700 1 -- start start 2024-01-18T06:53:20.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:20.283+0000 7f45ab59e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45ac0f78e0 0x7f45ac1064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:20.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:20.283+0000 7f45ab59e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ac1069d0 con 0x7f45ac0f78e0 2024-01-18T06:53:20.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:20.284+0000 7f45aa59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45ac0f78e0 0x7f45ac1064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:20.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:20.284+0000 7f45aa59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45ac0f78e0 0x7f45ac1064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:48164/0 (socket says 172.21.15.194:48164) 2024-01-18T06:53:20.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:20.284+0000 7f45aa59c700 1 -- 172.21.15.194:0/1038784506 learned_addr learned my addr 172.21.15.194:0/1038784506 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:53:20.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:20.284+0000 7f45aa59c700 1 -- 172.21.15.194:0/1038784506 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f459c005040 con 0x7f45ac0f78e0 2024-01-18T06:53:20.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:20.284+0000 7f45aa59c700 1 --2- 172.21.15.194:0/1038784506 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45ac0f78e0 0x7f45ac1064e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f459c013510 tx=0x7f459c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:20.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:20.284+0000 7f45a2ffd700 1 -- 172.21.15.194:0/1038784506 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f459c009320 con 0x7f45ac0f78e0 2024-01-18T06:53:20.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:20.284+0000 7f45a2ffd700 1 -- 172.21.15.194:0/1038784506 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f459c013f00 con 0x7f45ac0f78e0 2024-01-18T06:53:20.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:20.284+0000 7f45ab59e700 1 -- 172.21.15.194:0/1038784506 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45ac106bd0 con 0x7f45ac0f78e0 2024-01-18T06:53:20.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:20.284+0000 7f45a2ffd700 1 -- 172.21.15.194:0/1038784506 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f459c014dc0 con 0x7f45ac0f78e0 2024-01-18T06:53:20.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:20.284+0000 7f45ab59e700 1 -- 172.21.15.194:0/1038784506 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45ac107060 con 0x7f45ac0f78e0 2024-01-18T06:53:20.286 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:20.285+0000 7f45a2ffd700 1 -- 172.21.15.194:0/1038784506 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f459c0095f0 con 0x7f45ac0f78e0 2024-01-18T06:53:20.286 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:20.285+0000 7f45ab59e700 1 -- 172.21.15.194:0/1038784506 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45ac045ad0 con 0x7f45ac0f78e0 2024-01-18T06:53:20.286 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:20.285+0000 7f45a2ffd700 1 --2- 172.21.15.194:0/1038784506 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f459802ff60 0x7f4598032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:20.286 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:20.285+0000 7f45a2ffd700 1 -- 172.21.15.194:0/1038784506 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f459c041000 con 0x7f45ac0f78e0 2024-01-18T06:53:20.286 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:20.286+0000 7f45a9d9b700 1 --2- 172.21.15.194:0/1038784506 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f459802ff60 0x7f4598032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:20.286 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:20.286+0000 7f45a9d9b700 1 --2- 172.21.15.194:0/1038784506 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f459802ff60 0x7f4598032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f45940096a0 tx=0x7f4594006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:20.289 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:20.290+0000 7f45a2ffd700 1 -- 172.21.15.194:0/1038784506 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f459c014420 con 0x7f45ac0f78e0 2024-01-18T06:53:20.496 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:20.496+0000 7f45ab59e700 1 -- 172.21.15.194:0/1038784506 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f45ac107f10 con 0x7f45ac0f78e0 2024-01-18T06:53:20.496 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:20.497+0000 7f45a2ffd700 1 -- 172.21.15.194:0/1038784506 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f459c010230 con 0x7f45ac0f78e0 2024-01-18T06:53:20.497 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:53:20.497 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:53:20.498 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:20.499+0000 7f45ab59e700 1 -- 172.21.15.194:0/1038784506 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f459802ff60 msgr2=0x7f4598032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:20.498 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:20.499+0000 7f45ab59e700 1 --2- 172.21.15.194:0/1038784506 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f459802ff60 0x7f4598032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f45940096a0 tx=0x7f4594006b40).stop 2024-01-18T06:53:20.499 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:20.499+0000 7f45ab59e700 1 -- 172.21.15.194:0/1038784506 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45ac0f78e0 msgr2=0x7f45ac1064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:20.499 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:20.499+0000 7f45ab59e700 1 --2- 172.21.15.194:0/1038784506 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45ac0f78e0 0x7f45ac1064e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f459c013510 tx=0x7f459c00b040).stop 2024-01-18T06:53:20.499 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:20.499+0000 7f45ab59e700 1 -- 172.21.15.194:0/1038784506 shutdown_connections 2024-01-18T06:53:20.499 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:20.499+0000 7f45ab59e700 1 --2- 172.21.15.194:0/1038784506 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f459802ff60 0x7f4598032420 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:20.499 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:20.499+0000 7f45ab59e700 1 --2- 172.21.15.194:0/1038784506 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f45ac0f78e0 0x7f45ac1064e0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:20.499 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:20.499+0000 7f45ab59e700 1 -- 172.21.15.194:0/1038784506 >> 172.21.15.194:0/1038784506 conn(0x7f45ac0f0120 msgr2=0x7f45ac0f0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:20.499 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:20.499+0000 7f45ab59e700 1 -- 172.21.15.194:0/1038784506 shutdown_connections 2024-01-18T06:53:20.499 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:20.499+0000 7f45ab59e700 1 -- 172.21.15.194:0/1038784506 wait complete. 2024-01-18T06:53:20.499 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:53:21.464 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:21 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:53:20.019154+0000 mgr.smithi047.fuhzzr (mgr.14162) 146 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:53:21.464 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:21 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:53:20.497772+0000 mon.smithi047 (mon.0) 217 : audit [DBG] from='client.? 172.21.15.194:0/1038784506' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:53:21.892 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:53:21.892 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:53:23.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:23.080+0000 7f3651df8700 1 -- 172.21.15.194:0/1401515882 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f364c0f47c0 msgr2=0x7f364c0f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:23.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:23.080+0000 7f3651df8700 1 --2- 172.21.15.194:0/1401515882 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f364c0f47c0 0x7f364c0f4be0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f3634009fb0 tx=0x7f3634009b30).stop 2024-01-18T06:53:23.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:23.080+0000 7f3651df8700 1 -- 172.21.15.194:0/1401515882 shutdown_connections 2024-01-18T06:53:23.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:23.080+0000 7f3651df8700 1 --2- 172.21.15.194:0/1401515882 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f364c0f47c0 0x7f364c0f4be0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:23.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:23.080+0000 7f3651df8700 1 -- 172.21.15.194:0/1401515882 >> 172.21.15.194:0/1401515882 conn(0x7f364c0f02e0 msgr2=0x7f364c0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:23.081 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:23.080+0000 7f3651df8700 1 -- 172.21.15.194:0/1401515882 shutdown_connections 2024-01-18T06:53:23.081 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:23.080+0000 7f3651df8700 1 -- 172.21.15.194:0/1401515882 wait complete. 2024-01-18T06:53:23.081 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:23.081+0000 7f3651df8700 1 Processor -- start 2024-01-18T06:53:23.081 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:23.081+0000 7f3651df8700 1 -- start start 2024-01-18T06:53:23.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:23.081+0000 7f3651df8700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f364c0f47c0 0x7f364c106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:23.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:23.081+0000 7f3651df8700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364c106bf0 con 0x7f364c0f47c0 2024-01-18T06:53:23.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:23.081+0000 7f364b7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f364c0f47c0 0x7f364c106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:23.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:23.081+0000 7f364b7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f364c0f47c0 0x7f364c106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:48182/0 (socket says 172.21.15.194:48182) 2024-01-18T06:53:23.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:23.081+0000 7f364b7fe700 1 -- 172.21.15.194:0/1238851686 learned_addr learned my addr 172.21.15.194:0/1238851686 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:53:23.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:23.082+0000 7f364b7fe700 1 -- 172.21.15.194:0/1238851686 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3634005040 con 0x7f364c0f47c0 2024-01-18T06:53:23.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:23.082+0000 7f364b7fe700 1 --2- 172.21.15.194:0/1238851686 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f364c0f47c0 0x7f364c106700 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f3634013510 tx=0x7f363400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:23.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:23.082+0000 7f3643fff700 1 -- 172.21.15.194:0/1238851686 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3634009320 con 0x7f364c0f47c0 2024-01-18T06:53:23.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:23.082+0000 7f3643fff700 1 -- 172.21.15.194:0/1238851686 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3634013f00 con 0x7f364c0f47c0 2024-01-18T06:53:23.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:23.082+0000 7f3651df8700 1 -- 172.21.15.194:0/1238851686 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f364c106df0 con 0x7f364c0f47c0 2024-01-18T06:53:23.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:23.082+0000 7f3643fff700 1 -- 172.21.15.194:0/1238851686 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3634014d90 con 0x7f364c0f47c0 2024-01-18T06:53:23.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:23.082+0000 7f3651df8700 1 -- 172.21.15.194:0/1238851686 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f364c107280 con 0x7f364c0f47c0 2024-01-18T06:53:23.084 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:23.083+0000 7f3643fff700 1 -- 172.21.15.194:0/1238851686 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f36340095f0 con 0x7f364c0f47c0 2024-01-18T06:53:23.084 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:23.083+0000 7f3651df8700 1 -- 172.21.15.194:0/1238851686 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f364c045ad0 con 0x7f364c0f47c0 2024-01-18T06:53:23.084 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:23.083+0000 7f3643fff700 1 --2- 172.21.15.194:0/1238851686 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f363802ff60 0x7f3638032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:23.084 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:23.083+0000 7f3643fff700 1 -- 172.21.15.194:0/1238851686 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3634041000 con 0x7f364c0f47c0 2024-01-18T06:53:23.084 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:23.083+0000 7f364affd700 1 --2- 172.21.15.194:0/1238851686 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f363802ff60 0x7f3638032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:23.084 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:23.084+0000 7f364affd700 1 --2- 172.21.15.194:0/1238851686 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f363802ff60 0x7f3638032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f363c0096a0 tx=0x7f363c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:23.087 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:23.088+0000 7f3643fff700 1 -- 172.21.15.194:0/1238851686 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f363401d7f0 con 0x7f364c0f47c0 2024-01-18T06:53:23.292 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:23.292+0000 7f3651df8700 1 -- 172.21.15.194:0/1238851686 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f364c108130 con 0x7f364c0f47c0 2024-01-18T06:53:23.292 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:23.293+0000 7f3643fff700 1 -- 172.21.15.194:0/1238851686 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3634010230 con 0x7f364c0f47c0 2024-01-18T06:53:23.292 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:53:23.293 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:53:23.294 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:23.294+0000 7f3651df8700 1 -- 172.21.15.194:0/1238851686 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f363802ff60 msgr2=0x7f3638032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:23.294 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:23.294+0000 7f3651df8700 1 --2- 172.21.15.194:0/1238851686 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f363802ff60 0x7f3638032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f363c0096a0 tx=0x7f363c006b40).stop 2024-01-18T06:53:23.294 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:23.294+0000 7f3651df8700 1 -- 172.21.15.194:0/1238851686 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f364c0f47c0 msgr2=0x7f364c106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:23.294 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:23.294+0000 7f3651df8700 1 --2- 172.21.15.194:0/1238851686 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f364c0f47c0 0x7f364c106700 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f3634013510 tx=0x7f363400b040).stop 2024-01-18T06:53:23.294 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:23.294+0000 7f3651df8700 1 -- 172.21.15.194:0/1238851686 shutdown_connections 2024-01-18T06:53:23.294 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:23.294+0000 7f3651df8700 1 --2- 172.21.15.194:0/1238851686 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f363802ff60 0x7f3638032420 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:23.295 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:23.294+0000 7f3651df8700 1 --2- 172.21.15.194:0/1238851686 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f364c0f47c0 0x7f364c106700 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:23.295 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:23.294+0000 7f3651df8700 1 -- 172.21.15.194:0/1238851686 >> 172.21.15.194:0/1238851686 conn(0x7f364c0f02e0 msgr2=0x7f364c0f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:23.295 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:23.295+0000 7f3651df8700 1 -- 172.21.15.194:0/1238851686 shutdown_connections 2024-01-18T06:53:23.295 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:23.295+0000 7f3651df8700 1 -- 172.21.15.194:0/1238851686 wait complete. 2024-01-18T06:53:23.295 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:53:23.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:23 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:53:22.019423+0000 mgr.smithi047.fuhzzr (mgr.14162) 147 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:53:24.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:24 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:53:23.293860+0000 mon.smithi047 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.194:0/1238851686' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:53:24.685 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:53:24.685 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:53:25.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:25 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:53:24.019758+0000 mgr.smithi047.fuhzzr (mgr.14162) 148 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:53:25.871 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:25.871+0000 7efea936f700 1 -- 172.21.15.194:0/123935105 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efea40f5410 msgr2=0x7efea40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:25.871 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:25.871+0000 7efea936f700 1 --2- 172.21.15.194:0/123935105 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efea40f5410 0x7efea40f5830 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7efe94009fb0 tx=0x7efe94009b30).stop 2024-01-18T06:53:25.871 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:25.871+0000 7efea936f700 1 -- 172.21.15.194:0/123935105 shutdown_connections 2024-01-18T06:53:25.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:25.871+0000 7efea936f700 1 --2- 172.21.15.194:0/123935105 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efea40f5410 0x7efea40f5830 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:25.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:25.871+0000 7efea936f700 1 -- 172.21.15.194:0/123935105 >> 172.21.15.194:0/123935105 conn(0x7efea40f0b70 msgr2=0x7efea40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:25.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:25.872+0000 7efea936f700 1 -- 172.21.15.194:0/123935105 shutdown_connections 2024-01-18T06:53:25.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:25.872+0000 7efea936f700 1 -- 172.21.15.194:0/123935105 wait complete. 2024-01-18T06:53:25.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:25.872+0000 7efea936f700 1 Processor -- start 2024-01-18T06:53:25.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:25.872+0000 7efea936f700 1 -- start start 2024-01-18T06:53:25.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:25.872+0000 7efea936f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efea40f5410 0x7efea41066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:25.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:25.872+0000 7efea936f700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea4106bd0 con 0x7efea40f5410 2024-01-18T06:53:25.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:25.873+0000 7efea2ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efea40f5410 0x7efea41066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:25.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:25.873+0000 7efea2ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efea40f5410 0x7efea41066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:48202/0 (socket says 172.21.15.194:48202) 2024-01-18T06:53:25.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:25.873+0000 7efea2ffd700 1 -- 172.21.15.194:0/309994368 learned_addr learned my addr 172.21.15.194:0/309994368 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:53:25.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:25.873+0000 7efea2ffd700 1 -- 172.21.15.194:0/309994368 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe94005040 con 0x7efea40f5410 2024-01-18T06:53:25.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:25.873+0000 7efea2ffd700 1 --2- 172.21.15.194:0/309994368 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efea40f5410 0x7efea41066e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7efe94013510 tx=0x7efe9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:25.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:25.873+0000 7efe8b7fe700 1 -- 172.21.15.194:0/309994368 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe94009320 con 0x7efea40f5410 2024-01-18T06:53:25.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:25.873+0000 7efea936f700 1 -- 172.21.15.194:0/309994368 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efea4106dd0 con 0x7efea40f5410 2024-01-18T06:53:25.875 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:25.873+0000 7efea936f700 1 -- 172.21.15.194:0/309994368 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efea4107260 con 0x7efea40f5410 2024-01-18T06:53:25.875 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:25.873+0000 7efe8b7fe700 1 -- 172.21.15.194:0/309994368 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efe94013f00 con 0x7efea40f5410 2024-01-18T06:53:25.875 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:25.873+0000 7efe8b7fe700 1 -- 172.21.15.194:0/309994368 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe94014d90 con 0x7efea40f5410 2024-01-18T06:53:25.875 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:25.874+0000 7efea936f700 1 -- 172.21.15.194:0/309994368 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efea4045ad0 con 0x7efea40f5410 2024-01-18T06:53:25.875 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:25.874+0000 7efe8b7fe700 1 -- 172.21.15.194:0/309994368 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efe940095f0 con 0x7efea40f5410 2024-01-18T06:53:25.875 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:25.874+0000 7efe8b7fe700 1 --2- 172.21.15.194:0/309994368 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7efe8c02ff60 0x7efe8c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:25.875 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:25.874+0000 7efe8b7fe700 1 -- 172.21.15.194:0/309994368 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efe940410c0 con 0x7efea40f5410 2024-01-18T06:53:25.875 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:25.874+0000 7efea27fc700 1 --2- 172.21.15.194:0/309994368 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7efe8c02ff60 0x7efe8c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:25.875 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:25.875+0000 7efea27fc700 1 --2- 172.21.15.194:0/309994368 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7efe8c02ff60 0x7efe8c032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7efe980096a0 tx=0x7efe98006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:25.878 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:25.879+0000 7efe8b7fe700 1 -- 172.21.15.194:0/309994368 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efe94014420 con 0x7efea40f5410 2024-01-18T06:53:26.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:26.083+0000 7efea936f700 1 -- 172.21.15.194:0/309994368 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efea4108050 con 0x7efea40f5410 2024-01-18T06:53:26.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:26.084+0000 7efe8b7fe700 1 -- 172.21.15.194:0/309994368 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efe94010230 con 0x7efea40f5410 2024-01-18T06:53:26.084 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:53:26.084 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:53:26.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:26.086+0000 7efea936f700 1 -- 172.21.15.194:0/309994368 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7efe8c02ff60 msgr2=0x7efe8c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:26.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:26.086+0000 7efea936f700 1 --2- 172.21.15.194:0/309994368 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7efe8c02ff60 0x7efe8c032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7efe980096a0 tx=0x7efe98006b40).stop 2024-01-18T06:53:26.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:26.086+0000 7efea936f700 1 -- 172.21.15.194:0/309994368 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efea40f5410 msgr2=0x7efea41066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:26.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:26.086+0000 7efea936f700 1 --2- 172.21.15.194:0/309994368 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efea40f5410 0x7efea41066e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7efe94013510 tx=0x7efe9400b040).stop 2024-01-18T06:53:26.086 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:26.086+0000 7efea936f700 1 -- 172.21.15.194:0/309994368 shutdown_connections 2024-01-18T06:53:26.086 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:26.086+0000 7efea936f700 1 --2- 172.21.15.194:0/309994368 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7efe8c02ff60 0x7efe8c032420 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:26.086 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:26.086+0000 7efea936f700 1 --2- 172.21.15.194:0/309994368 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efea40f5410 0x7efea41066e0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:26.086 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:26.086+0000 7efea936f700 1 -- 172.21.15.194:0/309994368 >> 172.21.15.194:0/309994368 conn(0x7efea40f0b70 msgr2=0x7efea405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:26.086 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:26.086+0000 7efea936f700 1 -- 172.21.15.194:0/309994368 shutdown_connections 2024-01-18T06:53:26.086 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:26.086+0000 7efea936f700 1 -- 172.21.15.194:0/309994368 wait complete. 2024-01-18T06:53:26.086 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:53:26.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:26 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:53:26.085008+0000 mon.smithi047 (mon.0) 219 : audit [DBG] from='client.? 172.21.15.194:0/309994368' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:53:27.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:27 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:53:26.020075+0000 mgr.smithi047.fuhzzr (mgr.14162) 149 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:53:27.470 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:53:27.471 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:53:28.693 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:28.692+0000 7f1610f20700 1 -- 172.21.15.194:0/1617714278 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f160c0f6510 msgr2=0x7f160c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:28.693 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:28.692+0000 7f1610f20700 1 --2- 172.21.15.194:0/1617714278 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f160c0f6510 0x7f160c0f6930 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f1600009fb0 tx=0x7f1600009b30).stop 2024-01-18T06:53:28.693 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:28.692+0000 7f1610f20700 1 -- 172.21.15.194:0/1617714278 shutdown_connections 2024-01-18T06:53:28.693 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:28.692+0000 7f1610f20700 1 --2- 172.21.15.194:0/1617714278 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f160c0f6510 0x7f160c0f6930 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:28.693 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:28.692+0000 7f1610f20700 1 -- 172.21.15.194:0/1617714278 >> 172.21.15.194:0/1617714278 conn(0x7f160c0f1c70 msgr2=0x7f160c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:28.694 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:28.692+0000 7f1610f20700 1 -- 172.21.15.194:0/1617714278 shutdown_connections 2024-01-18T06:53:28.694 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:28.692+0000 7f1610f20700 1 -- 172.21.15.194:0/1617714278 wait complete. 2024-01-18T06:53:28.694 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:28.693+0000 7f1610f20700 1 Processor -- start 2024-01-18T06:53:28.694 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:28.693+0000 7f1610f20700 1 -- start start 2024-01-18T06:53:28.694 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:28.693+0000 7f1610f20700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f160c0f6510 0x7f160c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:28.694 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:28.693+0000 7f1610f20700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f160c106c20 con 0x7f160c0f6510 2024-01-18T06:53:28.694 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:28.693+0000 7f160a59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f160c0f6510 0x7f160c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:28.694 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:28.693+0000 7f160a59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f160c0f6510 0x7f160c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:44728/0 (socket says 172.21.15.194:44728) 2024-01-18T06:53:28.694 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:28.693+0000 7f160a59c700 1 -- 172.21.15.194:0/3059791326 learned_addr learned my addr 172.21.15.194:0/3059791326 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:53:28.694 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:28.693+0000 7f160a59c700 1 -- 172.21.15.194:0/3059791326 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1600005040 con 0x7f160c0f6510 2024-01-18T06:53:28.694 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:28.693+0000 7f160a59c700 1 --2- 172.21.15.194:0/3059791326 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f160c0f6510 0x7f160c106730 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f1600014000 tx=0x7f160000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:28.694 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:28.694+0000 7f15faffd700 1 -- 172.21.15.194:0/3059791326 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16000095f0 con 0x7f160c0f6510 2024-01-18T06:53:28.694 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:28.694+0000 7f1610f20700 1 -- 172.21.15.194:0/3059791326 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f160c106e20 con 0x7f160c0f6510 2024-01-18T06:53:28.694 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:28.694+0000 7f15faffd700 1 -- 172.21.15.194:0/3059791326 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1600007e70 con 0x7f160c0f6510 2024-01-18T06:53:28.694 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:28.694+0000 7f15faffd700 1 -- 172.21.15.194:0/3059791326 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1600014ea0 con 0x7f160c0f6510 2024-01-18T06:53:28.695 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:28.694+0000 7f1610f20700 1 -- 172.21.15.194:0/3059791326 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f160c1072b0 con 0x7f160c0f6510 2024-01-18T06:53:28.695 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:28.694+0000 7f1610f20700 1 -- 172.21.15.194:0/3059791326 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f160c045ad0 con 0x7f160c0f6510 2024-01-18T06:53:28.695 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:28.694+0000 7f15faffd700 1 -- 172.21.15.194:0/3059791326 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1600025070 con 0x7f160c0f6510 2024-01-18T06:53:28.695 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:28.695+0000 7f15faffd700 1 --2- 172.21.15.194:0/3059791326 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f15f402ff60 0x7f15f4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:28.695 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:28.695+0000 7f15faffd700 1 -- 172.21.15.194:0/3059791326 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1600040970 con 0x7f160c0f6510 2024-01-18T06:53:28.696 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:28.695+0000 7f1609d9b700 1 --2- 172.21.15.194:0/3059791326 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f15f402ff60 0x7f15f4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:28.696 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:28.695+0000 7f1609d9b700 1 --2- 172.21.15.194:0/3059791326 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f15f402ff60 0x7f15f4032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f15fc0096a0 tx=0x7f15fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:28.699 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:28.699+0000 7f15faffd700 1 -- 172.21.15.194:0/3059791326 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f16000079c0 con 0x7f160c0f6510 2024-01-18T06:53:28.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:28.904+0000 7f1610f20700 1 -- 172.21.15.194:0/3059791326 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f160c0fbf60 con 0x7f160c0f6510 2024-01-18T06:53:28.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:28.905+0000 7f15faffd700 1 -- 172.21.15.194:0/3059791326 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f160000f300 con 0x7f160c0f6510 2024-01-18T06:53:28.905 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:53:28.905 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:53:28.906 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:28.907+0000 7f1610f20700 1 -- 172.21.15.194:0/3059791326 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f15f402ff60 msgr2=0x7f15f4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:28.906 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:28.907+0000 7f1610f20700 1 --2- 172.21.15.194:0/3059791326 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f15f402ff60 0x7f15f4032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f15fc0096a0 tx=0x7f15fc006b40).stop 2024-01-18T06:53:28.906 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:28.907+0000 7f1610f20700 1 -- 172.21.15.194:0/3059791326 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f160c0f6510 msgr2=0x7f160c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:28.906 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:28.907+0000 7f1610f20700 1 --2- 172.21.15.194:0/3059791326 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f160c0f6510 0x7f160c106730 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f1600014000 tx=0x7f160000b040).stop 2024-01-18T06:53:28.906 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:28.907+0000 7f1610f20700 1 -- 172.21.15.194:0/3059791326 shutdown_connections 2024-01-18T06:53:28.907 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:28.907+0000 7f1610f20700 1 --2- 172.21.15.194:0/3059791326 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f15f402ff60 0x7f15f4032420 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:28.907 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:28.907+0000 7f1610f20700 1 --2- 172.21.15.194:0/3059791326 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f160c0f6510 0x7f160c106730 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:28.907 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:28.907+0000 7f1610f20700 1 -- 172.21.15.194:0/3059791326 >> 172.21.15.194:0/3059791326 conn(0x7f160c0f1c70 msgr2=0x7f160c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:28.907 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:28.907+0000 7f1610f20700 1 -- 172.21.15.194:0/3059791326 shutdown_connections 2024-01-18T06:53:28.907 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:28.907+0000 7f1610f20700 1 -- 172.21.15.194:0/3059791326 wait complete. 2024-01-18T06:53:28.907 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:53:29.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:28 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:53:28.020415+0000 mgr.smithi047.fuhzzr (mgr.14162) 150 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:53:29.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:28 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:53:28.905993+0000 mon.smithi047 (mon.0) 220 : audit [DBG] from='client.? 172.21.15.194:0/3059791326' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:53:30.300 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:53:30.300 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:53:31.447 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:31.447+0000 7f1ce547d700 1 -- 172.21.15.194:0/2455433537 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ce00f51a0 msgr2=0x7f1ce00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:31.447 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:31.447+0000 7f1ce547d700 1 --2- 172.21.15.194:0/2455433537 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ce00f51a0 0x7f1ce00f55c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f1cd0009fb0 tx=0x7f1cd0009b30).stop 2024-01-18T06:53:31.447 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:31.447+0000 7f1ce547d700 1 -- 172.21.15.194:0/2455433537 shutdown_connections 2024-01-18T06:53:31.448 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:31.447+0000 7f1ce547d700 1 --2- 172.21.15.194:0/2455433537 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ce00f51a0 0x7f1ce00f55c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:31.448 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:31.447+0000 7f1ce547d700 1 -- 172.21.15.194:0/2455433537 >> 172.21.15.194:0/2455433537 conn(0x7f1ce00f09a0 msgr2=0x7f1ce00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:31.448 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:31.447+0000 7f1ce547d700 1 -- 172.21.15.194:0/2455433537 shutdown_connections 2024-01-18T06:53:31.448 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:31.447+0000 7f1ce547d700 1 -- 172.21.15.194:0/2455433537 wait complete. 2024-01-18T06:53:31.448 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:31.448+0000 7f1ce547d700 1 Processor -- start 2024-01-18T06:53:31.448 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:31.448+0000 7f1ce547d700 1 -- start start 2024-01-18T06:53:31.448 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:31.448+0000 7f1ce547d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ce00f51a0 0x7f1ce00fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:31.448 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:31.448+0000 7f1ce547d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce00fb5a0 con 0x7f1ce00f51a0 2024-01-18T06:53:31.450 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:31.448+0000 7f1cdffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ce00f51a0 0x7f1ce00fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:31.450 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:31.448+0000 7f1cdffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ce00f51a0 0x7f1ce00fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:44734/0 (socket says 172.21.15.194:44734) 2024-01-18T06:53:31.450 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:31.448+0000 7f1cdffff700 1 -- 172.21.15.194:0/1163327332 learned_addr learned my addr 172.21.15.194:0/1163327332 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:53:31.450 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:31.449+0000 7f1cdffff700 1 -- 172.21.15.194:0/1163327332 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd0005040 con 0x7f1ce00f51a0 2024-01-18T06:53:31.450 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:31.449+0000 7f1cdffff700 1 --2- 172.21.15.194:0/1163327332 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ce00f51a0 0x7f1ce00fd120 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f1cd0009830 tx=0x7f1cd000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:31.450 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:31.449+0000 7f1cdcff9700 1 -- 172.21.15.194:0/1163327332 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cd0009320 con 0x7f1ce00f51a0 2024-01-18T06:53:31.450 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:31.449+0000 7f1cdcff9700 1 -- 172.21.15.194:0/1163327332 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1cd00136c0 con 0x7f1ce00f51a0 2024-01-18T06:53:31.450 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:31.449+0000 7f1ce547d700 1 -- 172.21.15.194:0/1163327332 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ce00fb740 con 0x7f1ce00f51a0 2024-01-18T06:53:31.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:31.449+0000 7f1cdcff9700 1 -- 172.21.15.194:0/1163327332 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cd001ee90 con 0x7f1ce00f51a0 2024-01-18T06:53:31.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:31.449+0000 7f1ce547d700 1 -- 172.21.15.194:0/1163327332 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ce00fbbd0 con 0x7f1ce00f51a0 2024-01-18T06:53:31.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:31.450+0000 7f1cdcff9700 1 -- 172.21.15.194:0/1163327332 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1cd0014440 con 0x7f1ce00f51a0 2024-01-18T06:53:31.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:31.450+0000 7f1cdcff9700 1 --2- 172.21.15.194:0/1163327332 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f1cc802ff10 0x7f1cc80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:31.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:31.450+0000 7f1ce547d700 1 -- 172.21.15.194:0/1163327332 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ce0059100 con 0x7f1ce00f51a0 2024-01-18T06:53:31.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:31.450+0000 7f1cdcff9700 1 -- 172.21.15.194:0/1163327332 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1cd0040f10 con 0x7f1ce00f51a0 2024-01-18T06:53:31.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:31.450+0000 7f1cdf7fe700 1 --2- 172.21.15.194:0/1163327332 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f1cc802ff10 0x7f1cc80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:31.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:31.451+0000 7f1cdf7fe700 1 --2- 172.21.15.194:0/1163327332 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f1cc802ff10 0x7f1cc80323d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f1cd40096a0 tx=0x7f1cd4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:31.454 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:31.455+0000 7f1cdcff9700 1 -- 172.21.15.194:0/1163327332 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1cd00273f0 con 0x7f1ce00f51a0 2024-01-18T06:53:31.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:31 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:53:30.020729+0000 mgr.smithi047.fuhzzr (mgr.14162) 151 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:53:31.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:31 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:53:30.116621+0000 mon.smithi047 (mon.0) 221 : audit [INF] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.fuhzzr/mirror_snapshot_schedule"}]: dispatch 2024-01-18T06:53:31.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:31 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:53:30.117343+0000 mon.smithi047 (mon.0) 222 : audit [INF] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.fuhzzr/trash_purge_schedule"}]: dispatch 2024-01-18T06:53:31.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:31.661+0000 7f1ce547d700 1 -- 172.21.15.194:0/1163327332 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1ce00fabe0 con 0x7f1ce00f51a0 2024-01-18T06:53:31.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:31.661+0000 7f1cdcff9700 1 -- 172.21.15.194:0/1163327332 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1cd001c320 con 0x7f1ce00f51a0 2024-01-18T06:53:31.661 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:53:31.661 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:53:31.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:31.663+0000 7f1ce547d700 1 -- 172.21.15.194:0/1163327332 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f1cc802ff10 msgr2=0x7f1cc80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:31.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:31.663+0000 7f1ce547d700 1 --2- 172.21.15.194:0/1163327332 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f1cc802ff10 0x7f1cc80323d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f1cd40096a0 tx=0x7f1cd4006b40).stop 2024-01-18T06:53:31.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:31.663+0000 7f1ce547d700 1 -- 172.21.15.194:0/1163327332 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ce00f51a0 msgr2=0x7f1ce00fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:31.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:31.663+0000 7f1ce547d700 1 --2- 172.21.15.194:0/1163327332 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ce00f51a0 0x7f1ce00fd120 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f1cd0009830 tx=0x7f1cd000b040).stop 2024-01-18T06:53:31.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:31.663+0000 7f1ce547d700 1 -- 172.21.15.194:0/1163327332 shutdown_connections 2024-01-18T06:53:31.663 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:31.663+0000 7f1ce547d700 1 --2- 172.21.15.194:0/1163327332 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f1cc802ff10 0x7f1cc80323d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:31.663 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:31.663+0000 7f1ce547d700 1 --2- 172.21.15.194:0/1163327332 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ce00f51a0 0x7f1ce00fd120 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:31.663 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:31.663+0000 7f1ce547d700 1 -- 172.21.15.194:0/1163327332 >> 172.21.15.194:0/1163327332 conn(0x7f1ce00f09a0 msgr2=0x7f1ce005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:31.663 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:31.663+0000 7f1ce547d700 1 -- 172.21.15.194:0/1163327332 shutdown_connections 2024-01-18T06:53:31.663 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:31.663+0000 7f1ce547d700 1 -- 172.21.15.194:0/1163327332 wait complete. 2024-01-18T06:53:31.663 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:53:32.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:32 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:53:31.662255+0000 mon.smithi047 (mon.0) 223 : audit [DBG] from='client.? 172.21.15.194:0/1163327332' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:53:33.057 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:53:33.058 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:53:33.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:33 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:53:32.021050+0000 mgr.smithi047.fuhzzr (mgr.14162) 152 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:53:34.212 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:34.212+0000 7f3241748700 1 -- 172.21.15.194:0/2676187860 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f323c0f5410 msgr2=0x7f323c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:34.212 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:34.212+0000 7f3241748700 1 --2- 172.21.15.194:0/2676187860 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f323c0f5410 0x7f323c0f5830 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f3224009fb0 tx=0x7f3224009b30).stop 2024-01-18T06:53:34.213 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:34.212+0000 7f3241748700 1 -- 172.21.15.194:0/2676187860 shutdown_connections 2024-01-18T06:53:34.213 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:34.212+0000 7f3241748700 1 --2- 172.21.15.194:0/2676187860 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f323c0f5410 0x7f323c0f5830 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:34.213 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:34.212+0000 7f3241748700 1 -- 172.21.15.194:0/2676187860 >> 172.21.15.194:0/2676187860 conn(0x7f323c0f0b70 msgr2=0x7f323c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:34.213 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:34.212+0000 7f3241748700 1 -- 172.21.15.194:0/2676187860 shutdown_connections 2024-01-18T06:53:34.213 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:34.212+0000 7f3241748700 1 -- 172.21.15.194:0/2676187860 wait complete. 2024-01-18T06:53:34.213 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:34.213+0000 7f3241748700 1 Processor -- start 2024-01-18T06:53:34.213 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:34.213+0000 7f3241748700 1 -- start start 2024-01-18T06:53:34.213 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:34.213+0000 7f3241748700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f323c0f5410 0x7f323c109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:34.213 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:34.213+0000 7f3241748700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323c060240 con 0x7f323c0f5410 2024-01-18T06:53:34.215 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:34.213+0000 7f323affd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f323c0f5410 0x7f323c109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:34.215 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:34.213+0000 7f323affd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f323c0f5410 0x7f323c109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:44748/0 (socket says 172.21.15.194:44748) 2024-01-18T06:53:34.215 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:34.213+0000 7f323affd700 1 -- 172.21.15.194:0/3804442652 learned_addr learned my addr 172.21.15.194:0/3804442652 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:53:34.215 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:34.214+0000 7f323affd700 1 -- 172.21.15.194:0/3804442652 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3224005040 con 0x7f323c0f5410 2024-01-18T06:53:34.215 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:34.214+0000 7f323affd700 1 --2- 172.21.15.194:0/3804442652 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f323c0f5410 0x7f323c109400 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f3224013510 tx=0x7f322400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:34.215 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:34.214+0000 7f32337fe700 1 -- 172.21.15.194:0/3804442652 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3224009320 con 0x7f323c0f5410 2024-01-18T06:53:34.215 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:34.214+0000 7f3241748700 1 -- 172.21.15.194:0/3804442652 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f323c060440 con 0x7f323c0f5410 2024-01-18T06:53:34.215 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:34.214+0000 7f32337fe700 1 -- 172.21.15.194:0/3804442652 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3224013f00 con 0x7f323c0f5410 2024-01-18T06:53:34.215 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:34.214+0000 7f32337fe700 1 -- 172.21.15.194:0/3804442652 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3224014d90 con 0x7f323c0f5410 2024-01-18T06:53:34.215 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:34.214+0000 7f3241748700 1 -- 172.21.15.194:0/3804442652 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f323c109ab0 con 0x7f323c0f5410 2024-01-18T06:53:34.216 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:34.215+0000 7f32337fe700 1 -- 172.21.15.194:0/3804442652 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f32240095f0 con 0x7f323c0f5410 2024-01-18T06:53:34.216 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:34.215+0000 7f3241748700 1 -- 172.21.15.194:0/3804442652 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f323c045ad0 con 0x7f323c0f5410 2024-01-18T06:53:34.216 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:34.215+0000 7f32337fe700 1 --2- 172.21.15.194:0/3804442652 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f322802ff60 0x7f3228032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:34.216 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:34.215+0000 7f32337fe700 1 -- 172.21.15.194:0/3804442652 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3224040fe0 con 0x7f323c0f5410 2024-01-18T06:53:34.216 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:34.215+0000 7f323a7fc700 1 --2- 172.21.15.194:0/3804442652 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f322802ff60 0x7f3228032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:34.216 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:34.216+0000 7f323a7fc700 1 --2- 172.21.15.194:0/3804442652 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f322802ff60 0x7f3228032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f322c0096a0 tx=0x7f322c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:34.219 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:34.220+0000 7f32337fe700 1 -- 172.21.15.194:0/3804442652 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f322401d7f0 con 0x7f323c0f5410 2024-01-18T06:53:34.423 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:34.424+0000 7f3241748700 1 -- 172.21.15.194:0/3804442652 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f323c0fae60 con 0x7f323c0f5410 2024-01-18T06:53:34.424 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:34.425+0000 7f32337fe700 1 -- 172.21.15.194:0/3804442652 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3224010230 con 0x7f323c0f5410 2024-01-18T06:53:34.424 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:53:34.424 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:53:34.426 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:34.426+0000 7f3241748700 1 -- 172.21.15.194:0/3804442652 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f322802ff60 msgr2=0x7f3228032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:34.426 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:34.426+0000 7f3241748700 1 --2- 172.21.15.194:0/3804442652 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f322802ff60 0x7f3228032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f322c0096a0 tx=0x7f322c006b40).stop 2024-01-18T06:53:34.426 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:34.426+0000 7f3241748700 1 -- 172.21.15.194:0/3804442652 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f323c0f5410 msgr2=0x7f323c109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:34.426 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:34.426+0000 7f3241748700 1 --2- 172.21.15.194:0/3804442652 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f323c0f5410 0x7f323c109400 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f3224013510 tx=0x7f322400b040).stop 2024-01-18T06:53:34.426 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:34.426+0000 7f3241748700 1 -- 172.21.15.194:0/3804442652 shutdown_connections 2024-01-18T06:53:34.426 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:34.426+0000 7f3241748700 1 --2- 172.21.15.194:0/3804442652 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f322802ff60 0x7f3228032420 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:34.426 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:34.426+0000 7f3241748700 1 --2- 172.21.15.194:0/3804442652 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f323c0f5410 0x7f323c109400 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:34.426 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:34.426+0000 7f3241748700 1 -- 172.21.15.194:0/3804442652 >> 172.21.15.194:0/3804442652 conn(0x7f323c0f0b70 msgr2=0x7f323c0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:34.426 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:34.426+0000 7f3241748700 1 -- 172.21.15.194:0/3804442652 shutdown_connections 2024-01-18T06:53:34.426 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:34.426+0000 7f3241748700 1 -- 172.21.15.194:0/3804442652 wait complete. 2024-01-18T06:53:34.427 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:53:35.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:35 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:53:34.021365+0000 mgr.smithi047.fuhzzr (mgr.14162) 153 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:53:35.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:35 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:53:34.425586+0000 mon.smithi047 (mon.0) 224 : audit [DBG] from='client.? 172.21.15.194:0/3804442652' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:53:35.805 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:53:35.806 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:53:36.986 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:36.986+0000 7f3cadc5c700 1 -- 172.21.15.194:0/3885273854 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3ca80f6480 msgr2=0x7f3ca80f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:36.986 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:36.986+0000 7f3cadc5c700 1 --2- 172.21.15.194:0/3885273854 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3ca80f6480 0x7f3ca80f68a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f3c98009fb0 tx=0x7f3c98009b30).stop 2024-01-18T06:53:36.986 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:36.986+0000 7f3cadc5c700 1 -- 172.21.15.194:0/3885273854 shutdown_connections 2024-01-18T06:53:36.987 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:36.986+0000 7f3cadc5c700 1 --2- 172.21.15.194:0/3885273854 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3ca80f6480 0x7f3ca80f68a0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:36.987 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:36.986+0000 7f3cadc5c700 1 -- 172.21.15.194:0/3885273854 >> 172.21.15.194:0/3885273854 conn(0x7f3ca80f1c00 msgr2=0x7f3ca80f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:36.987 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:36.987+0000 7f3cadc5c700 1 -- 172.21.15.194:0/3885273854 shutdown_connections 2024-01-18T06:53:36.987 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:36.987+0000 7f3cadc5c700 1 -- 172.21.15.194:0/3885273854 wait complete. 2024-01-18T06:53:36.987 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:36.987+0000 7f3cadc5c700 1 Processor -- start 2024-01-18T06:53:36.989 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:36.987+0000 7f3cadc5c700 1 -- start start 2024-01-18T06:53:36.989 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:36.987+0000 7f3cadc5c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3ca80f6480 0x7f3ca806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:36.989 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:36.987+0000 7f3cadc5c700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca8069e20 con 0x7f3ca80f6480 2024-01-18T06:53:36.989 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:36.987+0000 7f3ca77fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3ca80f6480 0x7f3ca806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:36.989 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:36.988+0000 7f3ca77fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3ca80f6480 0x7f3ca806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:44766/0 (socket says 172.21.15.194:44766) 2024-01-18T06:53:36.989 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:36.988+0000 7f3ca77fe700 1 -- 172.21.15.194:0/382087458 learned_addr learned my addr 172.21.15.194:0/382087458 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:53:36.989 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:36.988+0000 7f3ca77fe700 1 -- 172.21.15.194:0/382087458 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c98005040 con 0x7f3ca80f6480 2024-01-18T06:53:36.989 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:36.988+0000 7f3ca77fe700 1 --2- 172.21.15.194:0/382087458 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3ca80f6480 0x7f3ca806b9a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f3c98013510 tx=0x7f3c9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:36.989 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:36.988+0000 7f3c8ffff700 1 -- 172.21.15.194:0/382087458 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c98009320 con 0x7f3ca80f6480 2024-01-18T06:53:36.989 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:36.988+0000 7f3cadc5c700 1 -- 172.21.15.194:0/382087458 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ca8069fc0 con 0x7f3ca80f6480 2024-01-18T06:53:36.989 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:36.988+0000 7f3c8ffff700 1 -- 172.21.15.194:0/382087458 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3c98013f00 con 0x7f3ca80f6480 2024-01-18T06:53:36.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:36.988+0000 7f3c8ffff700 1 -- 172.21.15.194:0/382087458 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c98014d90 con 0x7f3ca80f6480 2024-01-18T06:53:36.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:36.988+0000 7f3cadc5c700 1 -- 172.21.15.194:0/382087458 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ca806a450 con 0x7f3ca80f6480 2024-01-18T06:53:36.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:36.989+0000 7f3c8ffff700 1 -- 172.21.15.194:0/382087458 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3c98014420 con 0x7f3ca80f6480 2024-01-18T06:53:36.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:36.989+0000 7f3c8ffff700 1 --2- 172.21.15.194:0/382087458 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3c9002ff60 0x7f3c90032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:36.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:36.989+0000 7f3cadc5c700 1 -- 172.21.15.194:0/382087458 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ca8059100 con 0x7f3ca80f6480 2024-01-18T06:53:36.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:36.989+0000 7f3c8ffff700 1 -- 172.21.15.194:0/382087458 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3c98041080 con 0x7f3ca80f6480 2024-01-18T06:53:36.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:36.989+0000 7f3ca6ffd700 1 --2- 172.21.15.194:0/382087458 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3c9002ff60 0x7f3c90032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:36.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:36.990+0000 7f3ca6ffd700 1 --2- 172.21.15.194:0/382087458 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3c9002ff60 0x7f3c90032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f3c9c0096a0 tx=0x7f3c9c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:36.993 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:36.993+0000 7f3c8ffff700 1 -- 172.21.15.194:0/382087458 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3c9806d2b0 con 0x7f3ca80f6480 2024-01-18T06:53:37.199 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:37.199+0000 7f3cadc5c700 1 -- 172.21.15.194:0/382087458 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3ca80fbf40 con 0x7f3ca80f6480 2024-01-18T06:53:37.200 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:37.200+0000 7f3c8ffff700 1 -- 172.21.15.194:0/382087458 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3c9801dd50 con 0x7f3ca80f6480 2024-01-18T06:53:37.200 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:53:37.200 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:53:37.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:37.202+0000 7f3cadc5c700 1 -- 172.21.15.194:0/382087458 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3c9002ff60 msgr2=0x7f3c90032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:37.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:37.202+0000 7f3cadc5c700 1 --2- 172.21.15.194:0/382087458 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3c9002ff60 0x7f3c90032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f3c9c0096a0 tx=0x7f3c9c006b40).stop 2024-01-18T06:53:37.202 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:37.202+0000 7f3cadc5c700 1 -- 172.21.15.194:0/382087458 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3ca80f6480 msgr2=0x7f3ca806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:37.202 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:37.202+0000 7f3cadc5c700 1 --2- 172.21.15.194:0/382087458 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3ca80f6480 0x7f3ca806b9a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f3c98013510 tx=0x7f3c9800b040).stop 2024-01-18T06:53:37.202 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:37.202+0000 7f3cadc5c700 1 -- 172.21.15.194:0/382087458 shutdown_connections 2024-01-18T06:53:37.202 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:37.202+0000 7f3cadc5c700 1 --2- 172.21.15.194:0/382087458 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3c9002ff60 0x7f3c90032420 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:37.202 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:37.202+0000 7f3cadc5c700 1 --2- 172.21.15.194:0/382087458 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3ca80f6480 0x7f3ca806b9a0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:37.202 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:37.202+0000 7f3cadc5c700 1 -- 172.21.15.194:0/382087458 >> 172.21.15.194:0/382087458 conn(0x7f3ca80f1c00 msgr2=0x7f3ca80f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:37.202 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:37.202+0000 7f3cadc5c700 1 -- 172.21.15.194:0/382087458 shutdown_connections 2024-01-18T06:53:37.202 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:37.202+0000 7f3cadc5c700 1 -- 172.21.15.194:0/382087458 wait complete. 2024-01-18T06:53:37.202 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:53:37.214 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:53:36.021617+0000 mgr.smithi047.fuhzzr (mgr.14162) 154 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:53:37.214 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cephadm 2024-01-18T06:53:36.083586+0000 mgr.smithi047.fuhzzr (mgr.14162) 155 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi047 ... 2024-01-18T06:53:37.214 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-01-18T06:53:37.214 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-18T06:53:37.214 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: stat: stderr time="2024-01-18T06:49:14Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T06:53:37.214 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: stat: stderr time="2024-01-18T06:49:52Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T06:53:37.214 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: stat: stderr time="2024-01-18T06:50:29Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T06:53:37.215 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-18T06:53:37.215 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-01-18T06:53:37.215 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-18T06:53:37.215 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: stat: stderr time="2024-01-18T06:51:43Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T06:53:37.215 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: stat: stderr time="2024-01-18T06:52:21Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T06:53:37.215 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: stat: stderr time="2024-01-18T06:52:58Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T06:53:37.215 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-18T06:53:37.215 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: Traceback (most recent call last): 2024-01-18T06:53:37.215 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: File "/var/lib/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-18T06:53:37.215 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: main() 2024-01-18T06:53:37.215 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: File "/var/lib/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-18T06:53:37.215 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: r = ctx.func(ctx) 2024-01-18T06:53:37.215 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: File "/var/lib/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-18T06:53:37.216 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: return func(ctx) 2024-01-18T06:53:37.216 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: File "/var/lib/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-18T06:53:37.216 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-18T06:53:37.216 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: File "/var/lib/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-01-18T06:53:37.216 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-01-18T06:53:37.216 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: File "/var/lib/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-18T06:53:37.216 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: raise RuntimeError('uid/gid not found') 2024-01-18T06:53:37.216 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: RuntimeError: uid/gid not found 2024-01-18T06:53:37.216 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: Traceback (most recent call last): 2024-01-18T06:53:37.216 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-18T06:53:37.216 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: yield (conn, connr) 2024-01-18T06:53:37.216 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-18T06:53:37.216 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: code, '\n'.join(err))) 2024-01-18T06:53:37.217 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi047 ... 2024-01-18T06:53:37.217 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-01-18T06:53:37.217 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-18T06:53:37.217 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: stat: stderr time="2024-01-18T06:49:14Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T06:53:37.217 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: stat: stderr time="2024-01-18T06:49:52Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T06:53:37.217 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: stat: stderr time="2024-01-18T06:50:29Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T06:53:37.217 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-18T06:53:37.217 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi047 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-01-18T06:53:37.217 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-18T06:53:37.217 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: stat: stderr time="2024-01-18T06:51:43Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T06:53:37.217 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: stat: stderr time="2024-01-18T06:52:21Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T06:53:37.217 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: stat: stderr time="2024-01-18T06:52:58Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T06:53:37.218 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-18T06:53:37.218 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: Traceback (most recent call last): 2024-01-18T06:53:37.218 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: File "/var/lib/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-18T06:53:37.218 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: main() 2024-01-18T06:53:37.218 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: File "/var/lib/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-18T06:53:37.218 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: r = ctx.func(ctx) 2024-01-18T06:53:37.218 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: File "/var/lib/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-18T06:53:37.218 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: return func(ctx) 2024-01-18T06:53:37.218 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: File "/var/lib/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-18T06:53:37.218 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-18T06:53:37.218 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: File "/var/lib/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-01-18T06:53:37.218 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-01-18T06:53:37.218 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: File "/var/lib/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-18T06:53:37.218 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: raise RuntimeError('uid/gid not found') 2024-01-18T06:53:37.218 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: RuntimeError: uid/gid not found 2024-01-18T06:53:37.219 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:53:36.084886+0000 mon.smithi047 (mon.0) 225 : audit [INF] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi047", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-18T06:53:37.219 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:53:36.087236+0000 mon.smithi047 (mon.0) 226 : audit [INF] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi047", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-01-18T06:53:37.219 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:53:36.088035+0000 mon.smithi047 (mon.0) 227 : audit [DBG] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-18T06:53:37.219 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cephadm 2024-01-18T06:53:36.088691+0000 mgr.smithi047.fuhzzr (mgr.14162) 156 : cephadm [INF] Deploying daemon crash.smithi047 on smithi047 2024-01-18T06:53:38.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:38 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:53:37.200949+0000 mon.smithi047 (mon.0) 228 : audit [DBG] from='client.? 172.21.15.194:0/382087458' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:53:38.464 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:38 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:53:37.952426+0000 mon.smithi047 (mon.0) 229 : audit [INF] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' 2024-01-18T06:53:38.464 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:38 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:53:38.047751+0000 mon.smithi047 (mon.0) 230 : audit [INF] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' 2024-01-18T06:53:38.464 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:38 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:53:38.050036+0000 mon.smithi047 (mon.0) 231 : audit [INF] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' 2024-01-18T06:53:38.464 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:38 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:53:38.050482+0000 mon.smithi047 (mon.0) 232 : audit [INF] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-18T06:53:38.464 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:38 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:53:38.053213+0000 mon.smithi047 (mon.0) 233 : audit [INF] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' 2024-01-18T06:53:38.578 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:53:38.578 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:53:39.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:39 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:53:38.021864+0000 mgr.smithi047.fuhzzr (mgr.14162) 157 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:53:39.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:39 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:53:38.050939+0000 mgr.smithi047.fuhzzr (mgr.14162) 158 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-18T06:53:39.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:39 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cephadm 2024-01-18T06:53:38.057860+0000 mgr.smithi047.fuhzzr (mgr.14162) 159 : cephadm [INF] Deploying daemon grafana.smithi047 on smithi047 2024-01-18T06:53:39.738 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:39.737+0000 7f803ec21700 1 -- 172.21.15.194:0/2836250210 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f80380f62a0 msgr2=0x7f80380f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:39.738 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:39.737+0000 7f803ec21700 1 --2- 172.21.15.194:0/2836250210 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f80380f62a0 0x7f80380f66c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f802c009fb0 tx=0x7f802c009b30).stop 2024-01-18T06:53:39.738 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:39.737+0000 7f803ec21700 1 -- 172.21.15.194:0/2836250210 shutdown_connections 2024-01-18T06:53:39.738 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:39.737+0000 7f803ec21700 1 --2- 172.21.15.194:0/2836250210 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f80380f62a0 0x7f80380f66c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:39.739 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:39.737+0000 7f803ec21700 1 -- 172.21.15.194:0/2836250210 >> 172.21.15.194:0/2836250210 conn(0x7f80380f1aa0 msgr2=0x7f80380f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:39.739 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:39.737+0000 7f803ec21700 1 -- 172.21.15.194:0/2836250210 shutdown_connections 2024-01-18T06:53:39.739 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:39.737+0000 7f803ec21700 1 -- 172.21.15.194:0/2836250210 wait complete. 2024-01-18T06:53:39.739 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:39.738+0000 7f803ec21700 1 Processor -- start 2024-01-18T06:53:39.739 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:39.738+0000 7f803ec21700 1 -- start start 2024-01-18T06:53:39.739 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:39.738+0000 7f803ec21700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f80380f62a0 0x7f803806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:39.739 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:39.738+0000 7f803ec21700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8038069e20 con 0x7f80380f62a0 2024-01-18T06:53:39.739 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:39.738+0000 7f803dc1f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f80380f62a0 0x7f803806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:39.739 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:39.738+0000 7f803dc1f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f80380f62a0 0x7f803806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:43780/0 (socket says 172.21.15.194:43780) 2024-01-18T06:53:39.739 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:39.738+0000 7f803dc1f700 1 -- 172.21.15.194:0/470618298 learned_addr learned my addr 172.21.15.194:0/470618298 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:53:39.739 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:39.738+0000 7f803dc1f700 1 -- 172.21.15.194:0/470618298 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f802c005040 con 0x7f80380f62a0 2024-01-18T06:53:39.739 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:39.738+0000 7f803dc1f700 1 --2- 172.21.15.194:0/470618298 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f80380f62a0 0x7f803806b9a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f802c013510 tx=0x7f802c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:39.739 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:39.738+0000 7f802a7fc700 1 -- 172.21.15.194:0/470618298 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f802c009320 con 0x7f80380f62a0 2024-01-18T06:53:39.739 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:39.739+0000 7f802a7fc700 1 -- 172.21.15.194:0/470618298 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f802c013f00 con 0x7f80380f62a0 2024-01-18T06:53:39.739 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:39.739+0000 7f803ec21700 1 -- 172.21.15.194:0/470618298 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8038069fc0 con 0x7f80380f62a0 2024-01-18T06:53:39.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:39.739+0000 7f802a7fc700 1 -- 172.21.15.194:0/470618298 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f802c014dc0 con 0x7f80380f62a0 2024-01-18T06:53:39.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:39.739+0000 7f803ec21700 1 -- 172.21.15.194:0/470618298 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f803806a450 con 0x7f80380f62a0 2024-01-18T06:53:39.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:39.739+0000 7f802a7fc700 1 -- 172.21.15.194:0/470618298 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f802c01d3d0 con 0x7f80380f62a0 2024-01-18T06:53:39.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:39.740+0000 7f802a7fc700 1 --2- 172.21.15.194:0/470618298 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f802402ff60 0x7f8024032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:39.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:39.740+0000 7f803ec21700 1 -- 172.21.15.194:0/470618298 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8038059100 con 0x7f80380f62a0 2024-01-18T06:53:39.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:39.740+0000 7f802a7fc700 1 -- 172.21.15.194:0/470618298 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f802c041100 con 0x7f80380f62a0 2024-01-18T06:53:39.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:39.740+0000 7f803d41e700 1 --2- 172.21.15.194:0/470618298 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f802402ff60 0x7f8024032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:39.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:39.740+0000 7f803d41e700 1 --2- 172.21.15.194:0/470618298 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f802402ff60 0x7f8024032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f80340096a0 tx=0x7f8034006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:39.744 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:39.744+0000 7f802a7fc700 1 -- 172.21.15.194:0/470618298 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f802c012920 con 0x7f80380f62a0 2024-01-18T06:53:39.948 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:39.949+0000 7f803ec21700 1 -- 172.21.15.194:0/470618298 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f80380fbce0 con 0x7f80380f62a0 2024-01-18T06:53:39.949 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:39.949+0000 7f802a7fc700 1 -- 172.21.15.194:0/470618298 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f802c012330 con 0x7f80380f62a0 2024-01-18T06:53:39.949 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:53:39.949 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:53:39.951 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:39.951+0000 7f803ec21700 1 -- 172.21.15.194:0/470618298 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f802402ff60 msgr2=0x7f8024032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:39.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:39.951+0000 7f803ec21700 1 --2- 172.21.15.194:0/470618298 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f802402ff60 0x7f8024032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f80340096a0 tx=0x7f8034006b40).stop 2024-01-18T06:53:39.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:39.951+0000 7f803ec21700 1 -- 172.21.15.194:0/470618298 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f80380f62a0 msgr2=0x7f803806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:39.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:39.951+0000 7f803ec21700 1 --2- 172.21.15.194:0/470618298 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f80380f62a0 0x7f803806b9a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f802c013510 tx=0x7f802c00b040).stop 2024-01-18T06:53:39.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:39.951+0000 7f803ec21700 1 -- 172.21.15.194:0/470618298 shutdown_connections 2024-01-18T06:53:39.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:39.951+0000 7f803ec21700 1 --2- 172.21.15.194:0/470618298 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f802402ff60 0x7f8024032420 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:39.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:39.951+0000 7f803ec21700 1 --2- 172.21.15.194:0/470618298 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f80380f62a0 0x7f803806b9a0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:39.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:39.951+0000 7f803ec21700 1 -- 172.21.15.194:0/470618298 >> 172.21.15.194:0/470618298 conn(0x7f80380f1aa0 msgr2=0x7f80380f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:39.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:39.951+0000 7f803ec21700 1 -- 172.21.15.194:0/470618298 shutdown_connections 2024-01-18T06:53:39.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:39.951+0000 7f803ec21700 1 -- 172.21.15.194:0/470618298 wait complete. 2024-01-18T06:53:39.953 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:53:40.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:40 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:53:39.950387+0000 mon.smithi047 (mon.0) 234 : audit [DBG] from='client.? 172.21.15.194:0/470618298' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:53:41.325 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:53:41.325 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:53:41.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:41 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:53:40.022203+0000 mgr.smithi047.fuhzzr (mgr.14162) 160 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:53:42.512 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:42.511+0000 7f9bae4bd700 1 -- 172.21.15.194:0/4109335038 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ba80601e0 msgr2=0x7f9ba80f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:42.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:42.511+0000 7f9bae4bd700 1 --2- 172.21.15.194:0/4109335038 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ba80601e0 0x7f9ba80f9190 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f9b98009fb0 tx=0x7f9b98009b30).stop 2024-01-18T06:53:42.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:42.511+0000 7f9bae4bd700 1 -- 172.21.15.194:0/4109335038 shutdown_connections 2024-01-18T06:53:42.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:42.511+0000 7f9bae4bd700 1 --2- 172.21.15.194:0/4109335038 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ba80601e0 0x7f9ba80f9190 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:42.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:42.511+0000 7f9bae4bd700 1 -- 172.21.15.194:0/4109335038 >> 172.21.15.194:0/4109335038 conn(0x7f9ba80f0b70 msgr2=0x7f9ba80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:42.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:42.511+0000 7f9bae4bd700 1 -- 172.21.15.194:0/4109335038 shutdown_connections 2024-01-18T06:53:42.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:42.511+0000 7f9bae4bd700 1 -- 172.21.15.194:0/4109335038 wait complete. 2024-01-18T06:53:42.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:42.512+0000 7f9bae4bd700 1 Processor -- start 2024-01-18T06:53:42.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:42.512+0000 7f9bae4bd700 1 -- start start 2024-01-18T06:53:42.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:42.512+0000 7f9bae4bd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ba80601e0 0x7f9ba81066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:42.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:42.512+0000 7f9bae4bd700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba8106bd0 con 0x7f9ba80601e0 2024-01-18T06:53:42.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:42.512+0000 7f9ba7fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ba80601e0 0x7f9ba81066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:42.514 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:42.512+0000 7f9ba7fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ba80601e0 0x7f9ba81066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:43804/0 (socket says 172.21.15.194:43804) 2024-01-18T06:53:42.514 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:42.512+0000 7f9ba7fff700 1 -- 172.21.15.194:0/3997964247 learned_addr learned my addr 172.21.15.194:0/3997964247 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:53:42.514 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:42.513+0000 7f9ba7fff700 1 -- 172.21.15.194:0/3997964247 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b98005040 con 0x7f9ba80601e0 2024-01-18T06:53:42.514 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:42.513+0000 7f9ba7fff700 1 --2- 172.21.15.194:0/3997964247 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ba80601e0 0x7f9ba81066e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f9b98004dd0 tx=0x7f9b98009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:42.514 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:42.513+0000 7f9ba4ff9700 1 -- 172.21.15.194:0/3997964247 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b98018070 con 0x7f9ba80601e0 2024-01-18T06:53:42.514 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:42.513+0000 7f9bae4bd700 1 -- 172.21.15.194:0/3997964247 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ba8106dd0 con 0x7f9ba80601e0 2024-01-18T06:53:42.514 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:42.513+0000 7f9ba4ff9700 1 -- 172.21.15.194:0/3997964247 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9b98013410 con 0x7f9ba80601e0 2024-01-18T06:53:42.514 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:42.513+0000 7f9bae4bd700 1 -- 172.21.15.194:0/3997964247 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ba8107260 con 0x7f9ba80601e0 2024-01-18T06:53:42.515 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:42.513+0000 7f9ba4ff9700 1 -- 172.21.15.194:0/3997964247 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b98027440 con 0x7f9ba80601e0 2024-01-18T06:53:42.515 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:42.514+0000 7f9ba4ff9700 1 -- 172.21.15.194:0/3997964247 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9b9801d420 con 0x7f9ba80601e0 2024-01-18T06:53:42.515 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:42.514+0000 7f9bae4bd700 1 -- 172.21.15.194:0/3997964247 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ba8045ad0 con 0x7f9ba80601e0 2024-01-18T06:53:42.515 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:42.514+0000 7f9ba4ff9700 1 --2- 172.21.15.194:0/3997964247 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f9b9002ff60 0x7f9b90032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:42.515 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:42.514+0000 7f9ba4ff9700 1 -- 172.21.15.194:0/3997964247 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9b98041410 con 0x7f9ba80601e0 2024-01-18T06:53:42.515 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:42.514+0000 7f9ba77fe700 1 --2- 172.21.15.194:0/3997964247 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f9b9002ff60 0x7f9b90032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:42.515 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:42.515+0000 7f9ba77fe700 1 --2- 172.21.15.194:0/3997964247 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f9b9002ff60 0x7f9b90032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f9b9c0096a0 tx=0x7f9b9c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:42.518 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:42.518+0000 7f9ba4ff9700 1 -- 172.21.15.194:0/3997964247 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9b9801d820 con 0x7f9ba80601e0 2024-01-18T06:53:42.726 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:42.726+0000 7f9bae4bd700 1 -- 172.21.15.194:0/3997964247 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9ba81080b0 con 0x7f9ba80601e0 2024-01-18T06:53:42.726 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:42.727+0000 7f9ba4ff9700 1 -- 172.21.15.194:0/3997964247 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9b98004490 con 0x7f9ba80601e0 2024-01-18T06:53:42.727 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:53:42.727 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:53:42.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:42.729+0000 7f9bae4bd700 1 -- 172.21.15.194:0/3997964247 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f9b9002ff60 msgr2=0x7f9b90032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:42.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:42.729+0000 7f9bae4bd700 1 --2- 172.21.15.194:0/3997964247 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f9b9002ff60 0x7f9b90032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f9b9c0096a0 tx=0x7f9b9c006b40).stop 2024-01-18T06:53:42.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:42.729+0000 7f9bae4bd700 1 -- 172.21.15.194:0/3997964247 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ba80601e0 msgr2=0x7f9ba81066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:42.729 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:42.729+0000 7f9bae4bd700 1 --2- 172.21.15.194:0/3997964247 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ba80601e0 0x7f9ba81066e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f9b98004dd0 tx=0x7f9b98009b30).stop 2024-01-18T06:53:42.729 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:42.729+0000 7f9bae4bd700 1 -- 172.21.15.194:0/3997964247 shutdown_connections 2024-01-18T06:53:42.729 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:42.729+0000 7f9bae4bd700 1 --2- 172.21.15.194:0/3997964247 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f9b9002ff60 0x7f9b90032420 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:42.729 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:42.729+0000 7f9bae4bd700 1 --2- 172.21.15.194:0/3997964247 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ba80601e0 0x7f9ba81066e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:42.729 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:42.729+0000 7f9bae4bd700 1 -- 172.21.15.194:0/3997964247 >> 172.21.15.194:0/3997964247 conn(0x7f9ba80f0b70 msgr2=0x7f9ba80f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:42.729 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:42.729+0000 7f9bae4bd700 1 -- 172.21.15.194:0/3997964247 shutdown_connections 2024-01-18T06:53:42.729 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:42.729+0000 7f9bae4bd700 1 -- 172.21.15.194:0/3997964247 wait complete. 2024-01-18T06:53:42.729 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:53:43.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:43 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:53:42.022580+0000 mgr.smithi047.fuhzzr (mgr.14162) 161 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:53:43.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:43 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:53:42.728152+0000 mon.smithi047 (mon.0) 235 : audit [DBG] from='client.? 172.21.15.194:0/3997964247' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:53:44.123 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:53:44.123 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:53:45.297 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:45.297+0000 7f3e860b5700 1 -- 172.21.15.194:0/1179819742 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3e800f6510 msgr2=0x7f3e800f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:45.297 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:45.297+0000 7f3e860b5700 1 --2- 172.21.15.194:0/1179819742 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3e800f6510 0x7f3e800f6930 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f3e74009fb0 tx=0x7f3e74009b30).stop 2024-01-18T06:53:45.298 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:45.297+0000 7f3e860b5700 1 -- 172.21.15.194:0/1179819742 shutdown_connections 2024-01-18T06:53:45.298 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:45.297+0000 7f3e860b5700 1 --2- 172.21.15.194:0/1179819742 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3e800f6510 0x7f3e800f6930 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:45.298 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:45.297+0000 7f3e860b5700 1 -- 172.21.15.194:0/1179819742 >> 172.21.15.194:0/1179819742 conn(0x7f3e800f1c70 msgr2=0x7f3e800f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:45.298 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:45.297+0000 7f3e860b5700 1 -- 172.21.15.194:0/1179819742 shutdown_connections 2024-01-18T06:53:45.298 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:45.297+0000 7f3e860b5700 1 -- 172.21.15.194:0/1179819742 wait complete. 2024-01-18T06:53:45.298 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:45.297+0000 7f3e860b5700 1 Processor -- start 2024-01-18T06:53:45.298 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:45.297+0000 7f3e860b5700 1 -- start start 2024-01-18T06:53:45.298 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:45.298+0000 7f3e860b5700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3e800f6510 0x7f3e8006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:45.298 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:45.298+0000 7f3e860b5700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e80069e20 con 0x7f3e800f6510 2024-01-18T06:53:45.298 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:45.298+0000 7f3e7f7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3e800f6510 0x7f3e8006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:45.298 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:45.298+0000 7f3e7f7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3e800f6510 0x7f3e8006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:43818/0 (socket says 172.21.15.194:43818) 2024-01-18T06:53:45.298 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:45.298+0000 7f3e7f7fe700 1 -- 172.21.15.194:0/994503517 learned_addr learned my addr 172.21.15.194:0/994503517 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:53:45.298 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:45.298+0000 7f3e7f7fe700 1 -- 172.21.15.194:0/994503517 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e74005040 con 0x7f3e800f6510 2024-01-18T06:53:45.298 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:45.298+0000 7f3e7f7fe700 1 --2- 172.21.15.194:0/994503517 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3e800f6510 0x7f3e8006b9a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f3e74013510 tx=0x7f3e7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:45.300 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:45.298+0000 7f3e67fff700 1 -- 172.21.15.194:0/994503517 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e74009320 con 0x7f3e800f6510 2024-01-18T06:53:45.300 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:45.298+0000 7f3e67fff700 1 -- 172.21.15.194:0/994503517 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3e74013f00 con 0x7f3e800f6510 2024-01-18T06:53:45.300 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:45.298+0000 7f3e860b5700 1 -- 172.21.15.194:0/994503517 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e80069fc0 con 0x7f3e800f6510 2024-01-18T06:53:45.300 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:45.298+0000 7f3e67fff700 1 -- 172.21.15.194:0/994503517 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e74014d90 con 0x7f3e800f6510 2024-01-18T06:53:45.300 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:45.298+0000 7f3e860b5700 1 -- 172.21.15.194:0/994503517 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e8006a450 con 0x7f3e800f6510 2024-01-18T06:53:45.300 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:45.299+0000 7f3e67fff700 1 -- 172.21.15.194:0/994503517 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3e74014420 con 0x7f3e800f6510 2024-01-18T06:53:45.300 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:45.299+0000 7f3e860b5700 1 -- 172.21.15.194:0/994503517 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e80059100 con 0x7f3e800f6510 2024-01-18T06:53:45.300 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:45.299+0000 7f3e67fff700 1 --2- 172.21.15.194:0/994503517 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3e6802ff60 0x7f3e68032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:45.300 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:45.299+0000 7f3e67fff700 1 -- 172.21.15.194:0/994503517 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3e74041000 con 0x7f3e800f6510 2024-01-18T06:53:45.300 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:45.300+0000 7f3e7effd700 1 --2- 172.21.15.194:0/994503517 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3e6802ff60 0x7f3e68032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:45.300 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:45.300+0000 7f3e7effd700 1 --2- 172.21.15.194:0/994503517 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3e6802ff60 0x7f3e68032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f3e700096a0 tx=0x7f3e70006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:45.304 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:45.304+0000 7f3e67fff700 1 -- 172.21.15.194:0/994503517 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3e740095f0 con 0x7f3e800f6510 2024-01-18T06:53:45.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:45 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:53:44.022807+0000 mgr.smithi047.fuhzzr (mgr.14162) 162 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:53:45.509 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:45.509+0000 7f3e860b5700 1 -- 172.21.15.194:0/994503517 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3e800fbf60 con 0x7f3e800f6510 2024-01-18T06:53:45.509 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:45.510+0000 7f3e67fff700 1 -- 172.21.15.194:0/994503517 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3e7401db70 con 0x7f3e800f6510 2024-01-18T06:53:45.509 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:53:45.510 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:53:45.511 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:45.511+0000 7f3e860b5700 1 -- 172.21.15.194:0/994503517 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3e6802ff60 msgr2=0x7f3e68032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:45.511 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:45.511+0000 7f3e860b5700 1 --2- 172.21.15.194:0/994503517 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3e6802ff60 0x7f3e68032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f3e700096a0 tx=0x7f3e70006b40).stop 2024-01-18T06:53:45.511 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:45.511+0000 7f3e860b5700 1 -- 172.21.15.194:0/994503517 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3e800f6510 msgr2=0x7f3e8006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:45.511 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:45.511+0000 7f3e860b5700 1 --2- 172.21.15.194:0/994503517 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3e800f6510 0x7f3e8006b9a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f3e74013510 tx=0x7f3e7400b040).stop 2024-01-18T06:53:45.511 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:45.511+0000 7f3e860b5700 1 -- 172.21.15.194:0/994503517 shutdown_connections 2024-01-18T06:53:45.511 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:45.511+0000 7f3e860b5700 1 --2- 172.21.15.194:0/994503517 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3e6802ff60 0x7f3e68032420 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:45.511 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:45.511+0000 7f3e860b5700 1 --2- 172.21.15.194:0/994503517 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3e800f6510 0x7f3e8006b9a0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:45.511 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:45.511+0000 7f3e860b5700 1 -- 172.21.15.194:0/994503517 >> 172.21.15.194:0/994503517 conn(0x7f3e800f1c70 msgr2=0x7f3e800f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:45.511 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:45.511+0000 7f3e860b5700 1 -- 172.21.15.194:0/994503517 shutdown_connections 2024-01-18T06:53:45.511 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:45.511+0000 7f3e860b5700 1 -- 172.21.15.194:0/994503517 wait complete. 2024-01-18T06:53:45.512 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:53:46.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:46 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:53:45.510779+0000 mon.smithi047 (mon.0) 236 : audit [DBG] from='client.? 172.21.15.194:0/994503517' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:53:46.884 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:53:46.885 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:53:47.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:47 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:53:46.023004+0000 mgr.smithi047.fuhzzr (mgr.14162) 163 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:53:48.081 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:48.080+0000 7fc8b4b05700 1 -- 172.21.15.194:0/3465079134 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8b00f5410 msgr2=0x7fc8b00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:48.081 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:48.080+0000 7fc8b4b05700 1 --2- 172.21.15.194:0/3465079134 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8b00f5410 0x7fc8b00f5830 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7fc898009fb0 tx=0x7fc898009b30).stop 2024-01-18T06:53:48.081 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:48.080+0000 7fc8b4b05700 1 -- 172.21.15.194:0/3465079134 shutdown_connections 2024-01-18T06:53:48.081 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:48.080+0000 7fc8b4b05700 1 --2- 172.21.15.194:0/3465079134 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8b00f5410 0x7fc8b00f5830 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:48.081 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:48.080+0000 7fc8b4b05700 1 -- 172.21.15.194:0/3465079134 >> 172.21.15.194:0/3465079134 conn(0x7fc8b00f0b70 msgr2=0x7fc8b00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:48.081 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:48.080+0000 7fc8b4b05700 1 -- 172.21.15.194:0/3465079134 shutdown_connections 2024-01-18T06:53:48.081 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:48.080+0000 7fc8b4b05700 1 -- 172.21.15.194:0/3465079134 wait complete. 2024-01-18T06:53:48.081 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:48.080+0000 7fc8b4b05700 1 Processor -- start 2024-01-18T06:53:48.081 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:48.081+0000 7fc8b4b05700 1 -- start start 2024-01-18T06:53:48.081 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:48.081+0000 7fc8b4b05700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8b00f5410 0x7fc8b0106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:48.081 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:48.081+0000 7fc8b4b05700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b0106c10 con 0x7fc8b00f5410 2024-01-18T06:53:48.081 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:48.081+0000 7fc8ae59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8b00f5410 0x7fc8b0106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:48.082 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:48.081+0000 7fc8ae59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8b00f5410 0x7fc8b0106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:56102/0 (socket says 172.21.15.194:56102) 2024-01-18T06:53:48.082 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:48.081+0000 7fc8ae59c700 1 -- 172.21.15.194:0/2599616465 learned_addr learned my addr 172.21.15.194:0/2599616465 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:53:48.082 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:48.081+0000 7fc8ae59c700 1 -- 172.21.15.194:0/2599616465 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc898005040 con 0x7fc8b00f5410 2024-01-18T06:53:48.082 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:48.081+0000 7fc8ae59c700 1 --2- 172.21.15.194:0/2599616465 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8b00f5410 0x7fc8b0106720 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7fc898013510 tx=0x7fc89800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:48.082 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:48.081+0000 7fc8a6ffd700 1 -- 172.21.15.194:0/2599616465 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc898009320 con 0x7fc8b00f5410 2024-01-18T06:53:48.082 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:48.081+0000 7fc8a6ffd700 1 -- 172.21.15.194:0/2599616465 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc898013f00 con 0x7fc8b00f5410 2024-01-18T06:53:48.082 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:48.081+0000 7fc8b4b05700 1 -- 172.21.15.194:0/2599616465 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8b0106e10 con 0x7fc8b00f5410 2024-01-18T06:53:48.082 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:48.081+0000 7fc8a6ffd700 1 -- 172.21.15.194:0/2599616465 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc898014d90 con 0x7fc8b00f5410 2024-01-18T06:53:48.082 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:48.081+0000 7fc8b4b05700 1 -- 172.21.15.194:0/2599616465 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8b01072a0 con 0x7fc8b00f5410 2024-01-18T06:53:48.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:48.082+0000 7fc8a6ffd700 1 -- 172.21.15.194:0/2599616465 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc8980095f0 con 0x7fc8b00f5410 2024-01-18T06:53:48.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:48.082+0000 7fc8b4b05700 1 -- 172.21.15.194:0/2599616465 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8b0045ad0 con 0x7fc8b00f5410 2024-01-18T06:53:48.084 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:48.082+0000 7fc8a6ffd700 1 --2- 172.21.15.194:0/2599616465 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc89c02ff60 0x7fc89c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:48.084 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:48.083+0000 7fc8a6ffd700 1 -- 172.21.15.194:0/2599616465 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc898041000 con 0x7fc8b00f5410 2024-01-18T06:53:48.084 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:48.083+0000 7fc8add9b700 1 --2- 172.21.15.194:0/2599616465 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc89c02ff60 0x7fc89c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:48.084 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:48.083+0000 7fc8add9b700 1 --2- 172.21.15.194:0/2599616465 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc89c02ff60 0x7fc89c032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fc8a00096a0 tx=0x7fc8a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:48.087 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:48.088+0000 7fc8a6ffd700 1 -- 172.21.15.194:0/2599616465 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc89801d7f0 con 0x7fc8b00f5410 2024-01-18T06:53:48.293 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:48.294+0000 7fc8b4b05700 1 -- 172.21.15.194:0/2599616465 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc8b0108150 con 0x7fc8b00f5410 2024-01-18T06:53:48.294 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:48.294+0000 7fc8a6ffd700 1 -- 172.21.15.194:0/2599616465 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc898010230 con 0x7fc8b00f5410 2024-01-18T06:53:48.294 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:53:48.294 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:53:48.295 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:48.296+0000 7fc8b4b05700 1 -- 172.21.15.194:0/2599616465 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc89c02ff60 msgr2=0x7fc89c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:48.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:48.296+0000 7fc8b4b05700 1 --2- 172.21.15.194:0/2599616465 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc89c02ff60 0x7fc89c032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fc8a00096a0 tx=0x7fc8a0006b40).stop 2024-01-18T06:53:48.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:48.296+0000 7fc8b4b05700 1 -- 172.21.15.194:0/2599616465 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8b00f5410 msgr2=0x7fc8b0106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:48.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:48.296+0000 7fc8b4b05700 1 --2- 172.21.15.194:0/2599616465 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8b00f5410 0x7fc8b0106720 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7fc898013510 tx=0x7fc89800b040).stop 2024-01-18T06:53:48.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:48.296+0000 7fc8b4b05700 1 -- 172.21.15.194:0/2599616465 shutdown_connections 2024-01-18T06:53:48.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:48.296+0000 7fc8b4b05700 1 --2- 172.21.15.194:0/2599616465 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc89c02ff60 0x7fc89c032420 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:48.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:48.296+0000 7fc8b4b05700 1 --2- 172.21.15.194:0/2599616465 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8b00f5410 0x7fc8b0106720 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:48.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:48.296+0000 7fc8b4b05700 1 -- 172.21.15.194:0/2599616465 >> 172.21.15.194:0/2599616465 conn(0x7fc8b00f0b70 msgr2=0x7fc8b005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:48.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:48.296+0000 7fc8b4b05700 1 -- 172.21.15.194:0/2599616465 shutdown_connections 2024-01-18T06:53:48.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:48.296+0000 7fc8b4b05700 1 -- 172.21.15.194:0/2599616465 wait complete. 2024-01-18T06:53:48.296 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:53:49.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:48 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:53:48.023264+0000 mgr.smithi047.fuhzzr (mgr.14162) 164 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:53:49.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:48 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:53:48.295238+0000 mon.smithi047 (mon.0) 237 : audit [DBG] from='client.? 172.21.15.194:0/2599616465' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:53:49.676 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:53:49.676 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:53:50.863 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:50.862+0000 7fe6044e8700 1 -- 172.21.15.194:0/3256989628 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe5fc0f6510 msgr2=0x7fe5fc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:50.863 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:50.862+0000 7fe6044e8700 1 --2- 172.21.15.194:0/3256989628 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe5fc0f6510 0x7fe5fc0f6930 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7fe5ec009fb0 tx=0x7fe5ec009b30).stop 2024-01-18T06:53:50.863 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:50.862+0000 7fe6044e8700 1 -- 172.21.15.194:0/3256989628 shutdown_connections 2024-01-18T06:53:50.863 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:50.862+0000 7fe6044e8700 1 --2- 172.21.15.194:0/3256989628 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe5fc0f6510 0x7fe5fc0f6930 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:50.863 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:50.862+0000 7fe6044e8700 1 -- 172.21.15.194:0/3256989628 >> 172.21.15.194:0/3256989628 conn(0x7fe5fc0f1c70 msgr2=0x7fe5fc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:50.864 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:50.862+0000 7fe6044e8700 1 -- 172.21.15.194:0/3256989628 shutdown_connections 2024-01-18T06:53:50.864 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:50.862+0000 7fe6044e8700 1 -- 172.21.15.194:0/3256989628 wait complete. 2024-01-18T06:53:50.864 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:50.863+0000 7fe6044e8700 1 Processor -- start 2024-01-18T06:53:50.864 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:50.863+0000 7fe6044e8700 1 -- start start 2024-01-18T06:53:50.864 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:50.863+0000 7fe6044e8700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe5fc0f6510 0x7fe5fc106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:50.864 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:50.863+0000 7fe6044e8700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5fc106c10 con 0x7fe5fc0f6510 2024-01-18T06:53:50.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:50.863+0000 7fe602284700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe5fc0f6510 0x7fe5fc106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:50.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:50.863+0000 7fe602284700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe5fc0f6510 0x7fe5fc106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:56120/0 (socket says 172.21.15.194:56120) 2024-01-18T06:53:50.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:50.863+0000 7fe602284700 1 -- 172.21.15.194:0/3639486986 learned_addr learned my addr 172.21.15.194:0/3639486986 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:53:50.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:50.863+0000 7fe602284700 1 -- 172.21.15.194:0/3639486986 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5ec005040 con 0x7fe5fc0f6510 2024-01-18T06:53:50.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:50.864+0000 7fe602284700 1 --2- 172.21.15.194:0/3639486986 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe5fc0f6510 0x7fe5fc106720 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fe5ec014000 tx=0x7fe5ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:50.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:50.864+0000 7fe5f2ffd700 1 -- 172.21.15.194:0/3639486986 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5ec0095f0 con 0x7fe5fc0f6510 2024-01-18T06:53:50.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:50.864+0000 7fe6044e8700 1 -- 172.21.15.194:0/3639486986 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5fc106e10 con 0x7fe5fc0f6510 2024-01-18T06:53:50.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:50.864+0000 7fe5f2ffd700 1 -- 172.21.15.194:0/3639486986 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe5ec007e70 con 0x7fe5fc0f6510 2024-01-18T06:53:50.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:50.864+0000 7fe5f2ffd700 1 -- 172.21.15.194:0/3639486986 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5ec014ea0 con 0x7fe5fc0f6510 2024-01-18T06:53:50.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:50.864+0000 7fe6044e8700 1 -- 172.21.15.194:0/3639486986 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5fc1072a0 con 0x7fe5fc0f6510 2024-01-18T06:53:50.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:50.864+0000 7fe5f2ffd700 1 -- 172.21.15.194:0/3639486986 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe5ec0079c0 con 0x7fe5fc0f6510 2024-01-18T06:53:50.867 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:50.865+0000 7fe6044e8700 1 -- 172.21.15.194:0/3639486986 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5fc045ad0 con 0x7fe5fc0f6510 2024-01-18T06:53:50.867 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:50.865+0000 7fe5f2ffd700 1 --2- 172.21.15.194:0/3639486986 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fe5e802ffb0 0x7fe5e8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:50.867 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:50.865+0000 7fe5f2ffd700 1 -- 172.21.15.194:0/3639486986 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe5ec0408b0 con 0x7fe5fc0f6510 2024-01-18T06:53:50.867 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:50.865+0000 7fe601a83700 1 --2- 172.21.15.194:0/3639486986 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fe5e802ffb0 0x7fe5e8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:50.867 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:50.865+0000 7fe601a83700 1 --2- 172.21.15.194:0/3639486986 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fe5e802ffb0 0x7fe5e8032470 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fe5f80096a0 tx=0x7fe5f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:50.869 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:50.869+0000 7fe5f2ffd700 1 -- 172.21.15.194:0/3639486986 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe5ec00f950 con 0x7fe5fc0f6510 2024-01-18T06:53:51.076 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:51.076+0000 7fe6044e8700 1 -- 172.21.15.194:0/3639486986 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe5fc108150 con 0x7fe5fc0f6510 2024-01-18T06:53:51.077 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:51.076+0000 7fe5f2ffd700 1 -- 172.21.15.194:0/3639486986 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe5ec010230 con 0x7fe5fc0f6510 2024-01-18T06:53:51.077 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:53:51.077 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:53:51.078 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:51.078+0000 7fe6044e8700 1 -- 172.21.15.194:0/3639486986 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fe5e802ffb0 msgr2=0x7fe5e8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:51.078 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:51.078+0000 7fe6044e8700 1 --2- 172.21.15.194:0/3639486986 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fe5e802ffb0 0x7fe5e8032470 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fe5f80096a0 tx=0x7fe5f8006b40).stop 2024-01-18T06:53:51.078 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:51.078+0000 7fe6044e8700 1 -- 172.21.15.194:0/3639486986 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe5fc0f6510 msgr2=0x7fe5fc106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:51.079 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:51.078+0000 7fe6044e8700 1 --2- 172.21.15.194:0/3639486986 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe5fc0f6510 0x7fe5fc106720 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fe5ec014000 tx=0x7fe5ec00b040).stop 2024-01-18T06:53:51.079 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:51.078+0000 7fe6044e8700 1 -- 172.21.15.194:0/3639486986 shutdown_connections 2024-01-18T06:53:51.079 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:51.078+0000 7fe6044e8700 1 --2- 172.21.15.194:0/3639486986 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fe5e802ffb0 0x7fe5e8032470 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:51.079 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:51.078+0000 7fe6044e8700 1 --2- 172.21.15.194:0/3639486986 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe5fc0f6510 0x7fe5fc106720 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:51.079 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:51.078+0000 7fe6044e8700 1 -- 172.21.15.194:0/3639486986 >> 172.21.15.194:0/3639486986 conn(0x7fe5fc0f1c70 msgr2=0x7fe5fc0f27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:51.079 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:51.078+0000 7fe6044e8700 1 -- 172.21.15.194:0/3639486986 shutdown_connections 2024-01-18T06:53:51.079 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:51.078+0000 7fe6044e8700 1 -- 172.21.15.194:0/3639486986 wait complete. 2024-01-18T06:53:51.079 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:53:51.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:51 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:53:50.023567+0000 mgr.smithi047.fuhzzr (mgr.14162) 165 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:53:52.450 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:53:52.450 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:53:52.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:52 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:53:51.078362+0000 mon.smithi047 (mon.0) 238 : audit [DBG] from='client.? 172.21.15.194:0/3639486986' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:53:53.174 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:53 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:53:52.023863+0000 mgr.smithi047.fuhzzr (mgr.14162) 166 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:53:53.614 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:53.613+0000 7f758fdea700 1 -- 172.21.15.194:0/121398186 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75880f4740 msgr2=0x7f75880f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:53.614 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:53.613+0000 7f758fdea700 1 --2- 172.21.15.194:0/121398186 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75880f4740 0x7f75880f4b60 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f7578009fb0 tx=0x7f7578009b30).stop 2024-01-18T06:53:53.615 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:53.613+0000 7f758fdea700 1 -- 172.21.15.194:0/121398186 shutdown_connections 2024-01-18T06:53:53.615 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:53.613+0000 7f758fdea700 1 --2- 172.21.15.194:0/121398186 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75880f4740 0x7f75880f4b60 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:53.615 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:53.613+0000 7f758fdea700 1 -- 172.21.15.194:0/121398186 >> 172.21.15.194:0/121398186 conn(0x7f75880f0260 msgr2=0x7f75880f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:53.615 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:53.613+0000 7f758fdea700 1 -- 172.21.15.194:0/121398186 shutdown_connections 2024-01-18T06:53:53.615 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:53.613+0000 7f758fdea700 1 -- 172.21.15.194:0/121398186 wait complete. 2024-01-18T06:53:53.615 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:53.614+0000 7f758fdea700 1 Processor -- start 2024-01-18T06:53:53.615 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:53.614+0000 7f758fdea700 1 -- start start 2024-01-18T06:53:53.615 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:53.614+0000 7f758fdea700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75880f4740 0x7f75880f72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:53.615 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:53.614+0000 7f758fdea700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75880f5770 con 0x7f75880f4740 2024-01-18T06:53:53.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:53.614+0000 7f758db86700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75880f4740 0x7f75880f72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:53.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:53.614+0000 7f758db86700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75880f4740 0x7f75880f72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:56138/0 (socket says 172.21.15.194:56138) 2024-01-18T06:53:53.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:53.614+0000 7f758db86700 1 -- 172.21.15.194:0/2008352322 learned_addr learned my addr 172.21.15.194:0/2008352322 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:53:53.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:53.615+0000 7f758db86700 1 -- 172.21.15.194:0/2008352322 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7578005040 con 0x7f75880f4740 2024-01-18T06:53:53.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:53.615+0000 7f758db86700 1 --2- 172.21.15.194:0/2008352322 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75880f4740 0x7f75880f72f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f7578013510 tx=0x7f757800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:53.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:53.615+0000 7f757e7fc700 1 -- 172.21.15.194:0/2008352322 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7578009320 con 0x7f75880f4740 2024-01-18T06:53:53.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:53.615+0000 7f757e7fc700 1 -- 172.21.15.194:0/2008352322 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7578013f00 con 0x7f75880f4740 2024-01-18T06:53:53.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:53.615+0000 7f758fdea700 1 -- 172.21.15.194:0/2008352322 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75880f5910 con 0x7f75880f4740 2024-01-18T06:53:53.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:53.615+0000 7f757e7fc700 1 -- 172.21.15.194:0/2008352322 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7578014d90 con 0x7f75880f4740 2024-01-18T06:53:53.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:53.615+0000 7f758fdea700 1 -- 172.21.15.194:0/2008352322 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75880f5da0 con 0x7f75880f4740 2024-01-18T06:53:53.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:53.616+0000 7f757e7fc700 1 -- 172.21.15.194:0/2008352322 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f75780095f0 con 0x7f75880f4740 2024-01-18T06:53:53.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:53.616+0000 7f757e7fc700 1 --2- 172.21.15.194:0/2008352322 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f757402ff60 0x7f7574032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:53.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:53.616+0000 7f758fdea700 1 -- 172.21.15.194:0/2008352322 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7588059100 con 0x7f75880f4740 2024-01-18T06:53:53.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:53.616+0000 7f757e7fc700 1 -- 172.21.15.194:0/2008352322 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7578040fe0 con 0x7f75880f4740 2024-01-18T06:53:53.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:53.616+0000 7f758d385700 1 --2- 172.21.15.194:0/2008352322 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f757402ff60 0x7f7574032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:53.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:53.616+0000 7f758d385700 1 --2- 172.21.15.194:0/2008352322 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f757402ff60 0x7f7574032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f75840096a0 tx=0x7f7584006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:53.621 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:53.620+0000 7f757e7fc700 1 -- 172.21.15.194:0/2008352322 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f757806d2b0 con 0x7f75880f4740 2024-01-18T06:53:53.827 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:53.826+0000 7f758fdea700 1 -- 172.21.15.194:0/2008352322 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f75880fbf40 con 0x7f75880f4740 2024-01-18T06:53:53.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:53.832+0000 7f757e7fc700 1 -- 172.21.15.194:0/2008352322 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f757801d7f0 con 0x7f75880f4740 2024-01-18T06:53:53.832 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:53:53.832 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:53:53.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:53.833+0000 7f758fdea700 1 -- 172.21.15.194:0/2008352322 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f757402ff60 msgr2=0x7f7574032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:53.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:53.833+0000 7f758fdea700 1 --2- 172.21.15.194:0/2008352322 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f757402ff60 0x7f7574032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f75840096a0 tx=0x7f7584006b40).stop 2024-01-18T06:53:53.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:53.833+0000 7f758fdea700 1 -- 172.21.15.194:0/2008352322 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75880f4740 msgr2=0x7f75880f72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:53.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:53.833+0000 7f758fdea700 1 --2- 172.21.15.194:0/2008352322 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75880f4740 0x7f75880f72f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f7578013510 tx=0x7f757800b040).stop 2024-01-18T06:53:53.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:53.833+0000 7f758fdea700 1 -- 172.21.15.194:0/2008352322 shutdown_connections 2024-01-18T06:53:53.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:53.833+0000 7f758fdea700 1 --2- 172.21.15.194:0/2008352322 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f757402ff60 0x7f7574032420 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:53.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:53.833+0000 7f758fdea700 1 --2- 172.21.15.194:0/2008352322 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75880f4740 0x7f75880f72f0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:53.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:53.833+0000 7f758fdea700 1 -- 172.21.15.194:0/2008352322 >> 172.21.15.194:0/2008352322 conn(0x7f75880f0260 msgr2=0x7f75880f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:53.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:53.833+0000 7f758fdea700 1 -- 172.21.15.194:0/2008352322 shutdown_connections 2024-01-18T06:53:53.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:53.833+0000 7f758fdea700 1 -- 172.21.15.194:0/2008352322 wait complete. 2024-01-18T06:53:53.835 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:53:54.331 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:54 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:53:53.829949+0000 mon.smithi047 (mon.0) 239 : audit [DBG] from='client.? 172.21.15.194:0/2008352322' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:53:55.238 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:53:55.238 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:53:55.713 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:55 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:53:54.024229+0000 mgr.smithi047.fuhzzr (mgr.14162) 167 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:53:55.714 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:55 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:53:54.327118+0000 mon.smithi047 (mon.0) 240 : audit [INF] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' 2024-01-18T06:53:55.714 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:55 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cephadm 2024-01-18T06:53:54.328700+0000 mgr.smithi047.fuhzzr (mgr.14162) 168 : cephadm [INF] Deploying daemon node-exporter.smithi047 on smithi047 2024-01-18T06:53:56.393 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:56.392+0000 7f01ed7f6700 1 -- 172.21.15.194:0/1426641766 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01e80f62a0 msgr2=0x7f01e80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:56.393 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:56.392+0000 7f01ed7f6700 1 --2- 172.21.15.194:0/1426641766 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01e80f62a0 0x7f01e80f66c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f01d8009fb0 tx=0x7f01d8009b30).stop 2024-01-18T06:53:56.393 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:56.392+0000 7f01ed7f6700 1 -- 172.21.15.194:0/1426641766 shutdown_connections 2024-01-18T06:53:56.394 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:56.392+0000 7f01ed7f6700 1 --2- 172.21.15.194:0/1426641766 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01e80f62a0 0x7f01e80f66c0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:56.394 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:56.392+0000 7f01ed7f6700 1 -- 172.21.15.194:0/1426641766 >> 172.21.15.194:0/1426641766 conn(0x7f01e80f1aa0 msgr2=0x7f01e80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:56.394 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:56.392+0000 7f01ed7f6700 1 -- 172.21.15.194:0/1426641766 shutdown_connections 2024-01-18T06:53:56.394 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:56.392+0000 7f01ed7f6700 1 -- 172.21.15.194:0/1426641766 wait complete. 2024-01-18T06:53:56.394 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:56.393+0000 7f01ed7f6700 1 Processor -- start 2024-01-18T06:53:56.394 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:56.393+0000 7f01ed7f6700 1 -- start start 2024-01-18T06:53:56.394 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:56.393+0000 7f01ed7f6700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01e80f62a0 0x7f01e81064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:56.394 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:56.393+0000 7f01ed7f6700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e81069a0 con 0x7f01e80f62a0 2024-01-18T06:53:56.395 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:56.393+0000 7f01e7fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01e80f62a0 0x7f01e81064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:56.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:56.393+0000 7f01e7fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01e80f62a0 0x7f01e81064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:56168/0 (socket says 172.21.15.194:56168) 2024-01-18T06:53:56.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:56.393+0000 7f01e7fff700 1 -- 172.21.15.194:0/4173983783 learned_addr learned my addr 172.21.15.194:0/4173983783 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:53:56.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:56.393+0000 7f01e7fff700 1 -- 172.21.15.194:0/4173983783 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d8005040 con 0x7f01e80f62a0 2024-01-18T06:53:56.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:56.394+0000 7f01e7fff700 1 --2- 172.21.15.194:0/4173983783 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01e80f62a0 0x7f01e81064b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f01d8013510 tx=0x7f01d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:56.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:56.394+0000 7f01e4ff9700 1 -- 172.21.15.194:0/4173983783 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01d8009320 con 0x7f01e80f62a0 2024-01-18T06:53:56.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:56.394+0000 7f01ed7f6700 1 -- 172.21.15.194:0/4173983783 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01e8106ba0 con 0x7f01e80f62a0 2024-01-18T06:53:56.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:56.394+0000 7f01e4ff9700 1 -- 172.21.15.194:0/4173983783 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f01d8013f00 con 0x7f01e80f62a0 2024-01-18T06:53:56.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:56.394+0000 7f01e4ff9700 1 -- 172.21.15.194:0/4173983783 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01d8014dc0 con 0x7f01e80f62a0 2024-01-18T06:53:56.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:56.394+0000 7f01ed7f6700 1 -- 172.21.15.194:0/4173983783 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01e8107030 con 0x7f01e80f62a0 2024-01-18T06:53:56.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:56.395+0000 7f01e4ff9700 1 -- 172.21.15.194:0/4173983783 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f01d80095f0 con 0x7f01e80f62a0 2024-01-18T06:53:56.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:56.395+0000 7f01ed7f6700 1 -- 172.21.15.194:0/4173983783 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01e8045ad0 con 0x7f01e80f62a0 2024-01-18T06:53:56.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:56.395+0000 7f01e4ff9700 1 --2- 172.21.15.194:0/4173983783 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f01d402ff60 0x7f01d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:56.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:56.395+0000 7f01e4ff9700 1 -- 172.21.15.194:0/4173983783 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f01d8041000 con 0x7f01e80f62a0 2024-01-18T06:53:56.397 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:56.395+0000 7f01e77fe700 1 --2- 172.21.15.194:0/4173983783 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f01d402ff60 0x7f01d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:56.397 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:56.396+0000 7f01e77fe700 1 --2- 172.21.15.194:0/4173983783 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f01d402ff60 0x7f01d4032420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f01d00096a0 tx=0x7f01d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:56.400 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:56.400+0000 7f01e4ff9700 1 -- 172.21.15.194:0/4173983783 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f01d801c790 con 0x7f01e80f62a0 2024-01-18T06:53:56.611 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:56.610+0000 7f01ed7f6700 1 -- 172.21.15.194:0/4173983783 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f01e8107ee0 con 0x7f01e80f62a0 2024-01-18T06:53:56.611 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:56.611+0000 7f01e4ff9700 1 -- 172.21.15.194:0/4173983783 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f01d8010230 con 0x7f01e80f62a0 2024-01-18T06:53:56.611 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:53:56.611 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:53:56.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:56.612+0000 7f01ed7f6700 1 -- 172.21.15.194:0/4173983783 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f01d402ff60 msgr2=0x7f01d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:56.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:56.612+0000 7f01ed7f6700 1 --2- 172.21.15.194:0/4173983783 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f01d402ff60 0x7f01d4032420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f01d00096a0 tx=0x7f01d0006b40).stop 2024-01-18T06:53:56.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:56.612+0000 7f01ed7f6700 1 -- 172.21.15.194:0/4173983783 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01e80f62a0 msgr2=0x7f01e81064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:56.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:56.612+0000 7f01ed7f6700 1 --2- 172.21.15.194:0/4173983783 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01e80f62a0 0x7f01e81064b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f01d8013510 tx=0x7f01d800b040).stop 2024-01-18T06:53:56.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:56.612+0000 7f01ed7f6700 1 -- 172.21.15.194:0/4173983783 shutdown_connections 2024-01-18T06:53:56.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:56.612+0000 7f01ed7f6700 1 --2- 172.21.15.194:0/4173983783 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f01d402ff60 0x7f01d4032420 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:56.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:56.612+0000 7f01ed7f6700 1 --2- 172.21.15.194:0/4173983783 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01e80f62a0 0x7f01e81064b0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:56.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:56.612+0000 7f01ed7f6700 1 -- 172.21.15.194:0/4173983783 >> 172.21.15.194:0/4173983783 conn(0x7f01e80f1aa0 msgr2=0x7f01e80f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:56.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:56.612+0000 7f01ed7f6700 1 -- 172.21.15.194:0/4173983783 shutdown_connections 2024-01-18T06:53:56.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:56.612+0000 7f01ed7f6700 1 -- 172.21.15.194:0/4173983783 wait complete. 2024-01-18T06:53:56.614 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:53:57.713 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:57 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:53:56.024598+0000 mgr.smithi047.fuhzzr (mgr.14162) 169 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:53:57.713 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:57 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:53:56.612599+0000 mon.smithi047 (mon.0) 241 : audit [DBG] from='client.? 172.21.15.194:0/4173983783' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:53:58.029 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:53:58.030 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:53:59.179 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:59.178+0000 7f539b30e700 1 -- 172.21.15.194:0/2595114932 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f53940f6510 msgr2=0x7f53940f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:59.179 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:59.178+0000 7f539b30e700 1 --2- 172.21.15.194:0/2595114932 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f53940f6510 0x7f53940f6930 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f5390009fb0 tx=0x7f5390009b30).stop 2024-01-18T06:53:59.179 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:59.178+0000 7f539b30e700 1 -- 172.21.15.194:0/2595114932 shutdown_connections 2024-01-18T06:53:59.179 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:59.178+0000 7f539b30e700 1 --2- 172.21.15.194:0/2595114932 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f53940f6510 0x7f53940f6930 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:59.179 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:59.178+0000 7f539b30e700 1 -- 172.21.15.194:0/2595114932 >> 172.21.15.194:0/2595114932 conn(0x7f53940f1c70 msgr2=0x7f53940f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:59.179 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:59.178+0000 7f539b30e700 1 -- 172.21.15.194:0/2595114932 shutdown_connections 2024-01-18T06:53:59.179 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:59.178+0000 7f539b30e700 1 -- 172.21.15.194:0/2595114932 wait complete. 2024-01-18T06:53:59.179 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:59.178+0000 7f539b30e700 1 Processor -- start 2024-01-18T06:53:59.179 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:59.178+0000 7f539b30e700 1 -- start start 2024-01-18T06:53:59.180 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:59.178+0000 7f539b30e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f53940f6510 0x7f5394106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:59.180 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:59.178+0000 7f539b30e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5394106c20 con 0x7f53940f6510 2024-01-18T06:53:59.180 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:59.179+0000 7f53990aa700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f53940f6510 0x7f5394106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:59.180 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:59.179+0000 7f53990aa700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f53940f6510 0x7f5394106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:56122/0 (socket says 172.21.15.194:56122) 2024-01-18T06:53:59.180 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:59.179+0000 7f53990aa700 1 -- 172.21.15.194:0/81098631 learned_addr learned my addr 172.21.15.194:0/81098631 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:53:59.180 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:59.179+0000 7f53990aa700 1 -- 172.21.15.194:0/81098631 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5390005040 con 0x7f53940f6510 2024-01-18T06:53:59.180 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:59.179+0000 7f53990aa700 1 --2- 172.21.15.194:0/81098631 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f53940f6510 0x7f5394106730 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f5390013510 tx=0x7f539000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:59.180 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:59.179+0000 7f5389ffb700 1 -- 172.21.15.194:0/81098631 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5390009320 con 0x7f53940f6510 2024-01-18T06:53:59.180 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:59.179+0000 7f539b30e700 1 -- 172.21.15.194:0/81098631 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5394106e20 con 0x7f53940f6510 2024-01-18T06:53:59.180 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:59.179+0000 7f5389ffb700 1 -- 172.21.15.194:0/81098631 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5390013f00 con 0x7f53940f6510 2024-01-18T06:53:59.180 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:59.179+0000 7f5389ffb700 1 -- 172.21.15.194:0/81098631 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5390014d90 con 0x7f53940f6510 2024-01-18T06:53:59.180 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:59.179+0000 7f539b30e700 1 -- 172.21.15.194:0/81098631 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53941072b0 con 0x7f53940f6510 2024-01-18T06:53:59.181 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:59.180+0000 7f5389ffb700 1 -- 172.21.15.194:0/81098631 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f53900095f0 con 0x7f53940f6510 2024-01-18T06:53:59.181 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:59.180+0000 7f539b30e700 1 -- 172.21.15.194:0/81098631 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5394045ad0 con 0x7f53940f6510 2024-01-18T06:53:59.181 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:59.180+0000 7f5389ffb700 1 --2- 172.21.15.194:0/81098631 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f538002ff60 0x7f5380032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:53:59.181 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:59.180+0000 7f5389ffb700 1 -- 172.21.15.194:0/81098631 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5390041080 con 0x7f53940f6510 2024-01-18T06:53:59.182 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:59.181+0000 7f53988a9700 1 --2- 172.21.15.194:0/81098631 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f538002ff60 0x7f5380032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:53:59.182 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:59.181+0000 7f53988a9700 1 --2- 172.21.15.194:0/81098631 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f538002ff60 0x7f5380032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f53840096a0 tx=0x7f5384006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:53:59.184 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:59.185+0000 7f5389ffb700 1 -- 172.21.15.194:0/81098631 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f539001d7f0 con 0x7f53940f6510 2024-01-18T06:53:59.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:58 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:53:58.024951+0000 mgr.smithi047.fuhzzr (mgr.14162) 170 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:53:59.389 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:59.389+0000 7f539b30e700 1 -- 172.21.15.194:0/81098631 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f53940fbf60 con 0x7f53940f6510 2024-01-18T06:53:59.389 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:59.390+0000 7f5389ffb700 1 -- 172.21.15.194:0/81098631 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5390010230 con 0x7f53940f6510 2024-01-18T06:53:59.390 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:53:59.390 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:53:59.392 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:59.392+0000 7f539b30e700 1 -- 172.21.15.194:0/81098631 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f538002ff60 msgr2=0x7f5380032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:59.392 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:59.392+0000 7f539b30e700 1 --2- 172.21.15.194:0/81098631 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f538002ff60 0x7f5380032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f53840096a0 tx=0x7f5384006b40).stop 2024-01-18T06:53:59.392 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:59.392+0000 7f539b30e700 1 -- 172.21.15.194:0/81098631 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f53940f6510 msgr2=0x7f5394106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:53:59.392 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:59.392+0000 7f539b30e700 1 --2- 172.21.15.194:0/81098631 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f53940f6510 0x7f5394106730 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f5390013510 tx=0x7f539000b040).stop 2024-01-18T06:53:59.392 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:59.392+0000 7f539b30e700 1 -- 172.21.15.194:0/81098631 shutdown_connections 2024-01-18T06:53:59.392 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:59.392+0000 7f539b30e700 1 --2- 172.21.15.194:0/81098631 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f538002ff60 0x7f5380032420 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:59.392 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:59.392+0000 7f539b30e700 1 --2- 172.21.15.194:0/81098631 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f53940f6510 0x7f5394106730 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:53:59.392 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:59.392+0000 7f539b30e700 1 -- 172.21.15.194:0/81098631 >> 172.21.15.194:0/81098631 conn(0x7f53940f1c70 msgr2=0x7f53940f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:53:59.392 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:59.392+0000 7f539b30e700 1 -- 172.21.15.194:0/81098631 shutdown_connections 2024-01-18T06:53:59.392 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:53:59.392+0000 7f539b30e700 1 -- 172.21.15.194:0/81098631 wait complete. 2024-01-18T06:53:59.392 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:54:00.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:53:59 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:53:59.391102+0000 mon.smithi047 (mon.0) 242 : audit [DBG] from='client.? 172.21.15.194:0/81098631' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:54:00.769 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:54:00.769 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:54:01.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:00 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:54:00.025335+0000 mgr.smithi047.fuhzzr (mgr.14162) 171 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:54:01.937 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:01.937+0000 7f35b54ae700 1 -- 172.21.15.194:0/3982096163 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35b00f62a0 msgr2=0x7f35b00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:01.937 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:01.937+0000 7f35b54ae700 1 --2- 172.21.15.194:0/3982096163 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35b00f62a0 0x7f35b00f66c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f35a0009fb0 tx=0x7f35a0009b30).stop 2024-01-18T06:54:01.937 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:01.937+0000 7f35b54ae700 1 -- 172.21.15.194:0/3982096163 shutdown_connections 2024-01-18T06:54:01.937 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:01.937+0000 7f35b54ae700 1 --2- 172.21.15.194:0/3982096163 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35b00f62a0 0x7f35b00f66c0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:01.937 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:01.937+0000 7f35b54ae700 1 -- 172.21.15.194:0/3982096163 >> 172.21.15.194:0/3982096163 conn(0x7f35b00f1aa0 msgr2=0x7f35b00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:54:01.937 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:01.937+0000 7f35b54ae700 1 -- 172.21.15.194:0/3982096163 shutdown_connections 2024-01-18T06:54:01.937 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:01.937+0000 7f35b54ae700 1 -- 172.21.15.194:0/3982096163 wait complete. 2024-01-18T06:54:01.937 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:01.937+0000 7f35b54ae700 1 Processor -- start 2024-01-18T06:54:01.938 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:01.938+0000 7f35b54ae700 1 -- start start 2024-01-18T06:54:01.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:01.938+0000 7f35b54ae700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35b00f62a0 0x7f35b01064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:54:01.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:01.938+0000 7f35b54ae700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b01069a0 con 0x7f35b00f62a0 2024-01-18T06:54:01.940 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:01.938+0000 7f35affff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35b00f62a0 0x7f35b01064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:54:01.940 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:01.938+0000 7f35affff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35b00f62a0 0x7f35b01064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:56140/0 (socket says 172.21.15.194:56140) 2024-01-18T06:54:01.940 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:01.938+0000 7f35affff700 1 -- 172.21.15.194:0/2599266909 learned_addr learned my addr 172.21.15.194:0/2599266909 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:54:01.940 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:01.939+0000 7f35affff700 1 -- 172.21.15.194:0/2599266909 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a0005040 con 0x7f35b00f62a0 2024-01-18T06:54:01.940 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:01.939+0000 7f35affff700 1 --2- 172.21.15.194:0/2599266909 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35b00f62a0 0x7f35b01064b0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f35a0013510 tx=0x7f35a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:54:01.940 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:01.939+0000 7f35acff9700 1 -- 172.21.15.194:0/2599266909 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35a0009320 con 0x7f35b00f62a0 2024-01-18T06:54:01.940 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:01.939+0000 7f35acff9700 1 -- 172.21.15.194:0/2599266909 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f35a0013f00 con 0x7f35b00f62a0 2024-01-18T06:54:01.940 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:01.939+0000 7f35b54ae700 1 -- 172.21.15.194:0/2599266909 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35b0106ba0 con 0x7f35b00f62a0 2024-01-18T06:54:01.940 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:01.939+0000 7f35acff9700 1 -- 172.21.15.194:0/2599266909 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35a0014dc0 con 0x7f35b00f62a0 2024-01-18T06:54:01.940 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:01.939+0000 7f35b54ae700 1 -- 172.21.15.194:0/2599266909 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35b0107030 con 0x7f35b00f62a0 2024-01-18T06:54:01.940 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:01.940+0000 7f35acff9700 1 -- 172.21.15.194:0/2599266909 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f35a00095f0 con 0x7f35b00f62a0 2024-01-18T06:54:01.940 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:01.940+0000 7f35b54ae700 1 -- 172.21.15.194:0/2599266909 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35b0045ad0 con 0x7f35b00f62a0 2024-01-18T06:54:01.940 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:01.940+0000 7f35acff9700 1 --2- 172.21.15.194:0/2599266909 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f359802ff60 0x7f3598032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:54:01.941 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:01.940+0000 7f35acff9700 1 -- 172.21.15.194:0/2599266909 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f35a0041080 con 0x7f35b00f62a0 2024-01-18T06:54:01.941 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:01.940+0000 7f35af7fe700 1 --2- 172.21.15.194:0/2599266909 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f359802ff60 0x7f3598032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:54:01.941 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:01.941+0000 7f35af7fe700 1 --2- 172.21.15.194:0/2599266909 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f359802ff60 0x7f3598032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f35a40096a0 tx=0x7f35a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:54:01.944 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:01.945+0000 7f35acff9700 1 -- 172.21.15.194:0/2599266909 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f35a001c790 con 0x7f35b00f62a0 2024-01-18T06:54:02.154 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:02.155+0000 7f35b54ae700 1 -- 172.21.15.194:0/2599266909 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f35b0107ee0 con 0x7f35b00f62a0 2024-01-18T06:54:02.155 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:02.155+0000 7f35acff9700 1 -- 172.21.15.194:0/2599266909 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f35a0010230 con 0x7f35b00f62a0 2024-01-18T06:54:02.155 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:54:02.155 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:54:02.157 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:02.157+0000 7f35b54ae700 1 -- 172.21.15.194:0/2599266909 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f359802ff60 msgr2=0x7f3598032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:02.157 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:02.157+0000 7f35b54ae700 1 --2- 172.21.15.194:0/2599266909 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f359802ff60 0x7f3598032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f35a40096a0 tx=0x7f35a4006b40).stop 2024-01-18T06:54:02.157 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:02.157+0000 7f35b54ae700 1 -- 172.21.15.194:0/2599266909 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35b00f62a0 msgr2=0x7f35b01064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:02.157 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:02.157+0000 7f35b54ae700 1 --2- 172.21.15.194:0/2599266909 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35b00f62a0 0x7f35b01064b0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f35a0013510 tx=0x7f35a000b040).stop 2024-01-18T06:54:02.157 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:02.157+0000 7f35b54ae700 1 -- 172.21.15.194:0/2599266909 shutdown_connections 2024-01-18T06:54:02.157 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:02.157+0000 7f35b54ae700 1 --2- 172.21.15.194:0/2599266909 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f359802ff60 0x7f3598032420 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:02.157 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:02.157+0000 7f35b54ae700 1 --2- 172.21.15.194:0/2599266909 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35b00f62a0 0x7f35b01064b0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:02.157 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:02.157+0000 7f35b54ae700 1 -- 172.21.15.194:0/2599266909 >> 172.21.15.194:0/2599266909 conn(0x7f35b00f1aa0 msgr2=0x7f35b00f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:54:02.157 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:02.157+0000 7f35b54ae700 1 -- 172.21.15.194:0/2599266909 shutdown_connections 2024-01-18T06:54:02.157 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:02.157+0000 7f35b54ae700 1 -- 172.21.15.194:0/2599266909 wait complete. 2024-01-18T06:54:02.158 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:54:03.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:03 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:54:02.025704+0000 mgr.smithi047.fuhzzr (mgr.14162) 172 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:54:03.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:03 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:54:02.156445+0000 mon.smithi047 (mon.0) 243 : audit [DBG] from='client.? 172.21.15.194:0/2599266909' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:54:03.547 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:54:03.548 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:54:04.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:04.752+0000 7efd6cc83700 1 -- 172.21.15.194:0/4179759892 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd680f6510 msgr2=0x7efd680f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:04.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:04.752+0000 7efd6cc83700 1 --2- 172.21.15.194:0/4179759892 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd680f6510 0x7efd680f6930 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7efd58009fb0 tx=0x7efd58009b30).stop 2024-01-18T06:54:04.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:04.752+0000 7efd6cc83700 1 -- 172.21.15.194:0/4179759892 shutdown_connections 2024-01-18T06:54:04.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:04.752+0000 7efd6cc83700 1 --2- 172.21.15.194:0/4179759892 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd680f6510 0x7efd680f6930 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:04.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:04.752+0000 7efd6cc83700 1 -- 172.21.15.194:0/4179759892 >> 172.21.15.194:0/4179759892 conn(0x7efd680f1c70 msgr2=0x7efd680f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:54:04.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:04.752+0000 7efd6cc83700 1 -- 172.21.15.194:0/4179759892 shutdown_connections 2024-01-18T06:54:04.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:04.752+0000 7efd6cc83700 1 -- 172.21.15.194:0/4179759892 wait complete. 2024-01-18T06:54:04.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:04.753+0000 7efd6cc83700 1 Processor -- start 2024-01-18T06:54:04.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:04.753+0000 7efd6cc83700 1 -- start start 2024-01-18T06:54:04.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:04.753+0000 7efd6cc83700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd680f6510 0x7efd68106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:54:04.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:04.753+0000 7efd6cc83700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd68106c20 con 0x7efd680f6510 2024-01-18T06:54:04.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:04.753+0000 7efd6659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd680f6510 0x7efd68106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:54:04.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:04.753+0000 7efd6659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd680f6510 0x7efd68106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:56148/0 (socket says 172.21.15.194:56148) 2024-01-18T06:54:04.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:04.753+0000 7efd6659c700 1 -- 172.21.15.194:0/2380968965 learned_addr learned my addr 172.21.15.194:0/2380968965 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:54:04.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:04.753+0000 7efd6659c700 1 -- 172.21.15.194:0/2380968965 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd58005040 con 0x7efd680f6510 2024-01-18T06:54:04.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:04.754+0000 7efd6659c700 1 --2- 172.21.15.194:0/2380968965 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd680f6510 0x7efd68106730 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7efd58013510 tx=0x7efd5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:54:04.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:04.754+0000 7efd56ffd700 1 -- 172.21.15.194:0/2380968965 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd58009320 con 0x7efd680f6510 2024-01-18T06:54:04.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:04.754+0000 7efd6cc83700 1 -- 172.21.15.194:0/2380968965 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd68106e20 con 0x7efd680f6510 2024-01-18T06:54:04.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:04.754+0000 7efd56ffd700 1 -- 172.21.15.194:0/2380968965 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efd58013f00 con 0x7efd680f6510 2024-01-18T06:54:04.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:04.754+0000 7efd56ffd700 1 -- 172.21.15.194:0/2380968965 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd58014d90 con 0x7efd680f6510 2024-01-18T06:54:04.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:04.754+0000 7efd6cc83700 1 -- 172.21.15.194:0/2380968965 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd681072b0 con 0x7efd680f6510 2024-01-18T06:54:04.756 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:04.755+0000 7efd56ffd700 1 -- 172.21.15.194:0/2380968965 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efd580095f0 con 0x7efd680f6510 2024-01-18T06:54:04.756 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:04.755+0000 7efd6cc83700 1 -- 172.21.15.194:0/2380968965 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd68045ad0 con 0x7efd680f6510 2024-01-18T06:54:04.756 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:04.755+0000 7efd56ffd700 1 --2- 172.21.15.194:0/2380968965 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7efd5002ff60 0x7efd50032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:54:04.756 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:04.755+0000 7efd56ffd700 1 -- 172.21.15.194:0/2380968965 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efd580410c0 con 0x7efd680f6510 2024-01-18T06:54:04.756 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:04.755+0000 7efd65d9b700 1 --2- 172.21.15.194:0/2380968965 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7efd5002ff60 0x7efd50032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:54:04.756 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:04.755+0000 7efd65d9b700 1 --2- 172.21.15.194:0/2380968965 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7efd5002ff60 0x7efd50032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7efd5c0096a0 tx=0x7efd5c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:54:04.759 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:04.759+0000 7efd56ffd700 1 -- 172.21.15.194:0/2380968965 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efd58014420 con 0x7efd680f6510 2024-01-18T06:54:04.962 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:04.962+0000 7efd6cc83700 1 -- 172.21.15.194:0/2380968965 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efd680fbf60 con 0x7efd680f6510 2024-01-18T06:54:04.962 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:04.963+0000 7efd56ffd700 1 -- 172.21.15.194:0/2380968965 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efd58010230 con 0x7efd680f6510 2024-01-18T06:54:04.962 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:54:04.963 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:54:04.964 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:04.964+0000 7efd6cc83700 1 -- 172.21.15.194:0/2380968965 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7efd5002ff60 msgr2=0x7efd50032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:04.964 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:04.964+0000 7efd6cc83700 1 --2- 172.21.15.194:0/2380968965 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7efd5002ff60 0x7efd50032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7efd5c0096a0 tx=0x7efd5c006b40).stop 2024-01-18T06:54:04.964 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:04.964+0000 7efd6cc83700 1 -- 172.21.15.194:0/2380968965 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd680f6510 msgr2=0x7efd68106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:04.964 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:04.964+0000 7efd6cc83700 1 --2- 172.21.15.194:0/2380968965 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd680f6510 0x7efd68106730 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7efd58013510 tx=0x7efd5800b040).stop 2024-01-18T06:54:04.965 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:04.965+0000 7efd6cc83700 1 -- 172.21.15.194:0/2380968965 shutdown_connections 2024-01-18T06:54:04.965 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:04.965+0000 7efd6cc83700 1 --2- 172.21.15.194:0/2380968965 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7efd5002ff60 0x7efd50032420 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:04.965 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:04.965+0000 7efd6cc83700 1 --2- 172.21.15.194:0/2380968965 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd680f6510 0x7efd68106730 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:04.965 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:04.965+0000 7efd6cc83700 1 -- 172.21.15.194:0/2380968965 >> 172.21.15.194:0/2380968965 conn(0x7efd680f1c70 msgr2=0x7efd680f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:54:04.965 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:04.965+0000 7efd6cc83700 1 -- 172.21.15.194:0/2380968965 shutdown_connections 2024-01-18T06:54:04.965 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:04.965+0000 7efd6cc83700 1 -- 172.21.15.194:0/2380968965 wait complete. 2024-01-18T06:54:04.965 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:54:05.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:05 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:54:04.026129+0000 mgr.smithi047.fuhzzr (mgr.14162) 173 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:54:05.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:05 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:54:04.963979+0000 mon.smithi047 (mon.0) 244 : audit [DBG] from='client.? 172.21.15.194:0/2380968965' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:54:06.306 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:54:06.306 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:54:07.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:07 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:54:06.026422+0000 mgr.smithi047.fuhzzr (mgr.14162) 174 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:54:07.519 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:07.518+0000 7f041505e700 1 -- 172.21.15.194:0/269120583 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04100f5410 msgr2=0x7f04100f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:07.519 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:07.518+0000 7f041505e700 1 --2- 172.21.15.194:0/269120583 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04100f5410 0x7f04100f5830 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f0404009fb0 tx=0x7f0404009b30).stop 2024-01-18T06:54:07.519 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:07.518+0000 7f041505e700 1 -- 172.21.15.194:0/269120583 shutdown_connections 2024-01-18T06:54:07.519 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:07.518+0000 7f041505e700 1 --2- 172.21.15.194:0/269120583 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04100f5410 0x7f04100f5830 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:07.519 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:07.518+0000 7f041505e700 1 -- 172.21.15.194:0/269120583 >> 172.21.15.194:0/269120583 conn(0x7f04100f0b70 msgr2=0x7f04100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:54:07.519 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:07.518+0000 7f041505e700 1 -- 172.21.15.194:0/269120583 shutdown_connections 2024-01-18T06:54:07.519 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:07.518+0000 7f041505e700 1 -- 172.21.15.194:0/269120583 wait complete. 2024-01-18T06:54:07.519 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:07.519+0000 7f041505e700 1 Processor -- start 2024-01-18T06:54:07.520 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:07.519+0000 7f041505e700 1 -- start start 2024-01-18T06:54:07.520 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:07.519+0000 7f041505e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04100fd360 0x7f04100fb820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:54:07.520 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:07.519+0000 7f041505e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04100fd780 con 0x7f04100fd360 2024-01-18T06:54:07.520 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:07.519+0000 7f040ed9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04100fd360 0x7f04100fb820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:54:07.520 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:07.519+0000 7f040ed9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04100fd360 0x7f04100fb820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:53520/0 (socket says 172.21.15.194:53520) 2024-01-18T06:54:07.520 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:07.519+0000 7f040ed9d700 1 -- 172.21.15.194:0/1058602822 learned_addr learned my addr 172.21.15.194:0/1058602822 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:54:07.520 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:07.520+0000 7f040ed9d700 1 -- 172.21.15.194:0/1058602822 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0404005040 con 0x7f04100fd360 2024-01-18T06:54:07.520 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:07.520+0000 7f040ed9d700 1 --2- 172.21.15.194:0/1058602822 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04100fd360 0x7f04100fb820 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f0404009f80 tx=0x7f0404009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:54:07.521 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:07.520+0000 7f03f77fe700 1 -- 172.21.15.194:0/1058602822 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f04040136c0 con 0x7f04100fd360 2024-01-18T06:54:07.521 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:07.520+0000 7f041505e700 1 -- 172.21.15.194:0/1058602822 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04100fbd10 con 0x7f04100fd360 2024-01-18T06:54:07.522 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:07.520+0000 7f03f77fe700 1 -- 172.21.15.194:0/1058602822 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f04040095f0 con 0x7f04100fd360 2024-01-18T06:54:07.522 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:07.520+0000 7f03f77fe700 1 -- 172.21.15.194:0/1058602822 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0404014420 con 0x7f04100fd360 2024-01-18T06:54:07.522 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:07.520+0000 7f041505e700 1 -- 172.21.15.194:0/1058602822 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04100fc200 con 0x7f04100fd360 2024-01-18T06:54:07.522 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:07.521+0000 7f041505e700 1 -- 172.21.15.194:0/1058602822 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0410059100 con 0x7f04100fd360 2024-01-18T06:54:07.522 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:07.521+0000 7f03f77fe700 1 -- 172.21.15.194:0/1058602822 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0404025060 con 0x7f04100fd360 2024-01-18T06:54:07.522 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:07.521+0000 7f03f77fe700 1 --2- 172.21.15.194:0/1058602822 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f03f802ff60 0x7f03f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:54:07.522 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:07.521+0000 7f03f77fe700 1 -- 172.21.15.194:0/1058602822 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0404040e40 con 0x7f04100fd360 2024-01-18T06:54:07.522 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:07.521+0000 7f040e59c700 1 --2- 172.21.15.194:0/1058602822 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f03f802ff60 0x7f03f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:54:07.522 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:07.522+0000 7f040e59c700 1 --2- 172.21.15.194:0/1058602822 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f03f802ff60 0x7f03f8032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f04000096a0 tx=0x7f0400006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:54:07.525 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:07.526+0000 7f03f77fe700 1 -- 172.21.15.194:0/1058602822 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f040401cb60 con 0x7f04100fd360 2024-01-18T06:54:07.731 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:07.732+0000 7f041505e700 1 -- 172.21.15.194:0/1058602822 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f04100fc5f0 con 0x7f04100fd360 2024-01-18T06:54:07.732 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:07.732+0000 7f03f77fe700 1 -- 172.21.15.194:0/1058602822 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f040400f280 con 0x7f04100fd360 2024-01-18T06:54:07.732 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:54:07.732 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:54:07.734 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:07.734+0000 7f041505e700 1 -- 172.21.15.194:0/1058602822 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f03f802ff60 msgr2=0x7f03f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:07.734 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:07.734+0000 7f041505e700 1 --2- 172.21.15.194:0/1058602822 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f03f802ff60 0x7f03f8032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f04000096a0 tx=0x7f0400006b40).stop 2024-01-18T06:54:07.734 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:07.734+0000 7f041505e700 1 -- 172.21.15.194:0/1058602822 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04100fd360 msgr2=0x7f04100fb820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:07.734 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:07.734+0000 7f041505e700 1 --2- 172.21.15.194:0/1058602822 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04100fd360 0x7f04100fb820 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f0404009f80 tx=0x7f0404009b30).stop 2024-01-18T06:54:07.734 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:07.734+0000 7f041505e700 1 -- 172.21.15.194:0/1058602822 shutdown_connections 2024-01-18T06:54:07.734 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:07.734+0000 7f041505e700 1 --2- 172.21.15.194:0/1058602822 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f03f802ff60 0x7f03f8032420 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:07.734 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:07.734+0000 7f041505e700 1 --2- 172.21.15.194:0/1058602822 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04100fd360 0x7f04100fb820 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:07.734 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:07.734+0000 7f041505e700 1 -- 172.21.15.194:0/1058602822 >> 172.21.15.194:0/1058602822 conn(0x7f04100f0b70 msgr2=0x7f041005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:54:07.734 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:07.734+0000 7f041505e700 1 -- 172.21.15.194:0/1058602822 shutdown_connections 2024-01-18T06:54:07.734 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:07.734+0000 7f041505e700 1 -- 172.21.15.194:0/1058602822 wait complete. 2024-01-18T06:54:07.735 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:54:08.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:08 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:54:07.733411+0000 mon.smithi047 (mon.0) 245 : audit [DBG] from='client.? 172.21.15.194:0/1058602822' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:54:09.109 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:54:09.109 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:54:09.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:09 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:54:08.026755+0000 mgr.smithi047.fuhzzr (mgr.14162) 175 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:54:10.323 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:10.322+0000 7f71508fa700 1 -- 172.21.15.194:0/3628086943 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f714c0f5410 msgr2=0x7f714c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:10.324 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:10.322+0000 7f71508fa700 1 --2- 172.21.15.194:0/3628086943 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f714c0f5410 0x7f714c0f5830 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f7134009fb0 tx=0x7f7134009b30).stop 2024-01-18T06:54:10.324 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:10.322+0000 7f71508fa700 1 -- 172.21.15.194:0/3628086943 shutdown_connections 2024-01-18T06:54:10.324 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:10.322+0000 7f71508fa700 1 --2- 172.21.15.194:0/3628086943 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f714c0f5410 0x7f714c0f5830 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:10.324 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:10.322+0000 7f71508fa700 1 -- 172.21.15.194:0/3628086943 >> 172.21.15.194:0/3628086943 conn(0x7f714c0f0b70 msgr2=0x7f714c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:54:10.324 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:10.322+0000 7f71508fa700 1 -- 172.21.15.194:0/3628086943 shutdown_connections 2024-01-18T06:54:10.324 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:10.322+0000 7f71508fa700 1 -- 172.21.15.194:0/3628086943 wait complete. 2024-01-18T06:54:10.324 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:10.323+0000 7f71508fa700 1 Processor -- start 2024-01-18T06:54:10.324 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:10.323+0000 7f71508fa700 1 -- start start 2024-01-18T06:54:10.324 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:10.323+0000 7f71508fa700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f714c0f5410 0x7f714c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:54:10.324 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:10.323+0000 7f71508fa700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714c106c20 con 0x7f714c0f5410 2024-01-18T06:54:10.324 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:10.323+0000 7f714a59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f714c0f5410 0x7f714c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:54:10.325 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:10.323+0000 7f714a59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f714c0f5410 0x7f714c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:53530/0 (socket says 172.21.15.194:53530) 2024-01-18T06:54:10.325 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:10.323+0000 7f714a59c700 1 -- 172.21.15.194:0/3113516902 learned_addr learned my addr 172.21.15.194:0/3113516902 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:54:10.325 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:10.324+0000 7f714a59c700 1 -- 172.21.15.194:0/3113516902 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7134005040 con 0x7f714c0f5410 2024-01-18T06:54:10.325 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:10.324+0000 7f714a59c700 1 --2- 172.21.15.194:0/3113516902 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f714c0f5410 0x7f714c106730 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f7134014000 tx=0x7f713400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:54:10.325 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:10.324+0000 7f7142ffd700 1 -- 172.21.15.194:0/3113516902 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71340095f0 con 0x7f714c0f5410 2024-01-18T06:54:10.325 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:10.324+0000 7f7142ffd700 1 -- 172.21.15.194:0/3113516902 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7134007e70 con 0x7f714c0f5410 2024-01-18T06:54:10.325 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:10.324+0000 7f7142ffd700 1 -- 172.21.15.194:0/3113516902 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7134014ea0 con 0x7f714c0f5410 2024-01-18T06:54:10.325 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:10.324+0000 7f71508fa700 1 -- 172.21.15.194:0/3113516902 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f714c106e20 con 0x7f714c0f5410 2024-01-18T06:54:10.325 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:10.324+0000 7f71508fa700 1 -- 172.21.15.194:0/3113516902 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f714c1072b0 con 0x7f714c0f5410 2024-01-18T06:54:10.326 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:10.325+0000 7f7142ffd700 1 -- 172.21.15.194:0/3113516902 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7134025070 con 0x7f714c0f5410 2024-01-18T06:54:10.326 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:10.325+0000 7f71508fa700 1 -- 172.21.15.194:0/3113516902 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f714c045ad0 con 0x7f714c0f5410 2024-01-18T06:54:10.326 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:10.325+0000 7f7142ffd700 1 --2- 172.21.15.194:0/3113516902 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f713802ff60 0x7f7138032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:54:10.326 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:10.325+0000 7f7142ffd700 1 -- 172.21.15.194:0/3113516902 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7134040950 con 0x7f714c0f5410 2024-01-18T06:54:10.326 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:10.325+0000 7f7149d9b700 1 --2- 172.21.15.194:0/3113516902 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f713802ff60 0x7f7138032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:54:10.326 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:10.326+0000 7f7149d9b700 1 --2- 172.21.15.194:0/3113516902 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f713802ff60 0x7f7138032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f713c0096a0 tx=0x7f713c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:54:10.329 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:10.330+0000 7f7142ffd700 1 -- 172.21.15.194:0/3113516902 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f71340079c0 con 0x7f714c0f5410 2024-01-18T06:54:10.535 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:10.535+0000 7f71508fa700 1 -- 172.21.15.194:0/3113516902 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f714c0fae60 con 0x7f714c0f5410 2024-01-18T06:54:10.535 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:10.536+0000 7f7142ffd700 1 -- 172.21.15.194:0/3113516902 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f713400f300 con 0x7f714c0f5410 2024-01-18T06:54:10.536 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:54:10.536 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:54:10.537 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:10.538+0000 7f71508fa700 1 -- 172.21.15.194:0/3113516902 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f713802ff60 msgr2=0x7f7138032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:10.537 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:10.538+0000 7f71508fa700 1 --2- 172.21.15.194:0/3113516902 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f713802ff60 0x7f7138032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f713c0096a0 tx=0x7f713c006b40).stop 2024-01-18T06:54:10.537 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:10.538+0000 7f71508fa700 1 -- 172.21.15.194:0/3113516902 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f714c0f5410 msgr2=0x7f714c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:10.537 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:10.538+0000 7f71508fa700 1 --2- 172.21.15.194:0/3113516902 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f714c0f5410 0x7f714c106730 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f7134014000 tx=0x7f713400b040).stop 2024-01-18T06:54:10.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:10.538+0000 7f71508fa700 1 -- 172.21.15.194:0/3113516902 shutdown_connections 2024-01-18T06:54:10.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:10.538+0000 7f71508fa700 1 --2- 172.21.15.194:0/3113516902 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f713802ff60 0x7f7138032420 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:10.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:10.538+0000 7f71508fa700 1 --2- 172.21.15.194:0/3113516902 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f714c0f5410 0x7f714c106730 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:10.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:10.538+0000 7f71508fa700 1 -- 172.21.15.194:0/3113516902 >> 172.21.15.194:0/3113516902 conn(0x7f714c0f0b70 msgr2=0x7f714c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:54:10.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:10.538+0000 7f71508fa700 1 -- 172.21.15.194:0/3113516902 shutdown_connections 2024-01-18T06:54:10.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:10.538+0000 7f71508fa700 1 -- 172.21.15.194:0/3113516902 wait complete. 2024-01-18T06:54:10.538 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:54:11.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:11 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:54:10.027177+0000 mgr.smithi047.fuhzzr (mgr.14162) 176 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:54:11.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:11 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:54:10.537236+0000 mon.smithi047 (mon.0) 246 : audit [DBG] from='client.? 172.21.15.194:0/3113516902' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:54:11.938 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:54:11.938 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:54:13.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:13.166+0000 7f9780d46700 1 -- 172.21.15.194:0/997622495 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f977c060200 msgr2=0x7f977c0fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:13.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:13.166+0000 7f9780d46700 1 --2- 172.21.15.194:0/997622495 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f977c060200 0x7f977c0fb570 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f976c009fb0 tx=0x7f976c009b30).stop 2024-01-18T06:54:13.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:13.166+0000 7f9780d46700 1 -- 172.21.15.194:0/997622495 shutdown_connections 2024-01-18T06:54:13.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:13.166+0000 7f9780d46700 1 --2- 172.21.15.194:0/997622495 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f977c060200 0x7f977c0fb570 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:13.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:13.166+0000 7f9780d46700 1 -- 172.21.15.194:0/997622495 >> 172.21.15.194:0/997622495 conn(0x7f977c0f0b70 msgr2=0x7f977c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:54:13.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:13.166+0000 7f9780d46700 1 -- 172.21.15.194:0/997622495 shutdown_connections 2024-01-18T06:54:13.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:13.166+0000 7f9780d46700 1 -- 172.21.15.194:0/997622495 wait complete. 2024-01-18T06:54:13.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:13.167+0000 7f9780d46700 1 Processor -- start 2024-01-18T06:54:13.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:13.167+0000 7f9780d46700 1 -- start start 2024-01-18T06:54:13.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:13.167+0000 7f9780d46700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f977c060200 0x7f977c1088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:54:13.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:13.167+0000 7f9780d46700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f977c108dc0 con 0x7f977c060200 2024-01-18T06:54:13.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:13.167+0000 7f977a59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f977c060200 0x7f977c1088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:54:13.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:13.167+0000 7f977a59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f977c060200 0x7f977c1088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:53540/0 (socket says 172.21.15.194:53540) 2024-01-18T06:54:13.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:13.167+0000 7f977a59c700 1 -- 172.21.15.194:0/445462645 learned_addr learned my addr 172.21.15.194:0/445462645 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:54:13.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:13.167+0000 7f977a59c700 1 -- 172.21.15.194:0/445462645 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f976c005040 con 0x7f977c060200 2024-01-18T06:54:13.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:13.167+0000 7f977a59c700 1 --2- 172.21.15.194:0/445462645 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f977c060200 0x7f977c1088d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f976c013510 tx=0x7f976c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:54:13.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:13.168+0000 7f976affd700 1 -- 172.21.15.194:0/445462645 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f976c009320 con 0x7f977c060200 2024-01-18T06:54:13.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:13.168+0000 7f976affd700 1 -- 172.21.15.194:0/445462645 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f976c013f00 con 0x7f977c060200 2024-01-18T06:54:13.169 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:13.168+0000 7f9780d46700 1 -- 172.21.15.194:0/445462645 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f977c108fc0 con 0x7f977c060200 2024-01-18T06:54:13.169 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:13.168+0000 7f976affd700 1 -- 172.21.15.194:0/445462645 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f976c014dc0 con 0x7f977c060200 2024-01-18T06:54:13.170 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:13.168+0000 7f9780d46700 1 -- 172.21.15.194:0/445462645 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f977c109450 con 0x7f977c060200 2024-01-18T06:54:13.170 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:13.168+0000 7f976affd700 1 -- 172.21.15.194:0/445462645 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f976c0095f0 con 0x7f977c060200 2024-01-18T06:54:13.170 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:13.168+0000 7f9780d46700 1 -- 172.21.15.194:0/445462645 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f977c045ad0 con 0x7f977c060200 2024-01-18T06:54:13.170 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:13.169+0000 7f976affd700 1 --2- 172.21.15.194:0/445462645 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f976402ff60 0x7f9764032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:54:13.170 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:13.169+0000 7f976affd700 1 -- 172.21.15.194:0/445462645 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f976c041100 con 0x7f977c060200 2024-01-18T06:54:13.170 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:13.169+0000 7f9779d9b700 1 --2- 172.21.15.194:0/445462645 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f976402ff60 0x7f9764032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:54:13.170 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:13.169+0000 7f9779d9b700 1 --2- 172.21.15.194:0/445462645 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f976402ff60 0x7f9764032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f97700096a0 tx=0x7f9770006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:54:13.173 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:13.173+0000 7f976affd700 1 -- 172.21.15.194:0/445462645 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f976c014420 con 0x7f977c060200 2024-01-18T06:54:13.377 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:13.377+0000 7f9780d46700 1 -- 172.21.15.194:0/445462645 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f977c10a300 con 0x7f977c060200 2024-01-18T06:54:13.378 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:13.378+0000 7f976affd700 1 -- 172.21.15.194:0/445462645 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f976c010230 con 0x7f977c060200 2024-01-18T06:54:13.378 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:54:13.378 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:54:13.379 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:13.380+0000 7f9780d46700 1 -- 172.21.15.194:0/445462645 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f976402ff60 msgr2=0x7f9764032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:13.379 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:13.380+0000 7f9780d46700 1 --2- 172.21.15.194:0/445462645 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f976402ff60 0x7f9764032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f97700096a0 tx=0x7f9770006b40).stop 2024-01-18T06:54:13.379 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:13.380+0000 7f9780d46700 1 -- 172.21.15.194:0/445462645 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f977c060200 msgr2=0x7f977c1088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:13.379 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:13.380+0000 7f9780d46700 1 --2- 172.21.15.194:0/445462645 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f977c060200 0x7f977c1088d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f976c013510 tx=0x7f976c00b040).stop 2024-01-18T06:54:13.379 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:13.380+0000 7f9780d46700 1 -- 172.21.15.194:0/445462645 shutdown_connections 2024-01-18T06:54:13.379 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:13.380+0000 7f9780d46700 1 --2- 172.21.15.194:0/445462645 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f976402ff60 0x7f9764032420 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:13.380 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:13.380+0000 7f9780d46700 1 --2- 172.21.15.194:0/445462645 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f977c060200 0x7f977c1088d0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:13.380 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:13.380+0000 7f9780d46700 1 -- 172.21.15.194:0/445462645 >> 172.21.15.194:0/445462645 conn(0x7f977c0f0b70 msgr2=0x7f977c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:54:13.380 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:13.380+0000 7f9780d46700 1 -- 172.21.15.194:0/445462645 shutdown_connections 2024-01-18T06:54:13.380 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:13.380+0000 7f9780d46700 1 -- 172.21.15.194:0/445462645 wait complete. 2024-01-18T06:54:13.380 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:54:13.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:13 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:54:12.027526+0000 mgr.smithi047.fuhzzr (mgr.14162) 177 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:54:14.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:14 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:54:13.379219+0000 mon.smithi047 (mon.0) 247 : audit [DBG] from='client.? 172.21.15.194:0/445462645' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:54:14.760 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:54:14.761 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:54:15.464 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:15 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:54:14.027768+0000 mgr.smithi047.fuhzzr (mgr.14162) 178 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:54:15.950 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:15.949+0000 7f574839d700 1 -- 172.21.15.194:0/581285727 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57400f6510 msgr2=0x7f57400f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:15.950 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:15.949+0000 7f574839d700 1 --2- 172.21.15.194:0/581285727 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57400f6510 0x7f57400f6930 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f5734009fb0 tx=0x7f5734009b30).stop 2024-01-18T06:54:15.950 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:15.949+0000 7f574839d700 1 -- 172.21.15.194:0/581285727 shutdown_connections 2024-01-18T06:54:15.950 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:15.949+0000 7f574839d700 1 --2- 172.21.15.194:0/581285727 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57400f6510 0x7f57400f6930 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:15.950 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:15.949+0000 7f574839d700 1 -- 172.21.15.194:0/581285727 >> 172.21.15.194:0/581285727 conn(0x7f57400f1c70 msgr2=0x7f57400f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:54:15.950 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:15.950+0000 7f574839d700 1 -- 172.21.15.194:0/581285727 shutdown_connections 2024-01-18T06:54:15.950 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:15.950+0000 7f574839d700 1 -- 172.21.15.194:0/581285727 wait complete. 2024-01-18T06:54:15.950 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:15.950+0000 7f574839d700 1 Processor -- start 2024-01-18T06:54:15.950 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:15.950+0000 7f574839d700 1 -- start start 2024-01-18T06:54:15.951 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:15.950+0000 7f574839d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57400f6510 0x7f5740106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:54:15.951 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:15.950+0000 7f574839d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5740106c20 con 0x7f57400f6510 2024-01-18T06:54:15.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:15.951+0000 7f5746139700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57400f6510 0x7f5740106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:54:15.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:15.951+0000 7f5746139700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57400f6510 0x7f5740106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:53562/0 (socket says 172.21.15.194:53562) 2024-01-18T06:54:15.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:15.951+0000 7f5746139700 1 -- 172.21.15.194:0/3753741021 learned_addr learned my addr 172.21.15.194:0/3753741021 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:54:15.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:15.951+0000 7f5746139700 1 -- 172.21.15.194:0/3753741021 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5734005040 con 0x7f57400f6510 2024-01-18T06:54:15.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:15.951+0000 7f5746139700 1 --2- 172.21.15.194:0/3753741021 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57400f6510 0x7f5740106730 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f5734013510 tx=0x7f573400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:54:15.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:15.951+0000 7f5732ffd700 1 -- 172.21.15.194:0/3753741021 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5734009320 con 0x7f57400f6510 2024-01-18T06:54:15.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:15.951+0000 7f574839d700 1 -- 172.21.15.194:0/3753741021 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5740106e20 con 0x7f57400f6510 2024-01-18T06:54:15.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:15.952+0000 7f5732ffd700 1 -- 172.21.15.194:0/3753741021 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5734013f00 con 0x7f57400f6510 2024-01-18T06:54:15.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:15.952+0000 7f5732ffd700 1 -- 172.21.15.194:0/3753741021 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5734014d90 con 0x7f57400f6510 2024-01-18T06:54:15.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:15.952+0000 7f574839d700 1 -- 172.21.15.194:0/3753741021 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57401072b0 con 0x7f57400f6510 2024-01-18T06:54:15.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:15.952+0000 7f5732ffd700 1 -- 172.21.15.194:0/3753741021 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5734014420 con 0x7f57400f6510 2024-01-18T06:54:15.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:15.952+0000 7f574839d700 1 -- 172.21.15.194:0/3753741021 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5740045ad0 con 0x7f57400f6510 2024-01-18T06:54:15.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:15.953+0000 7f5732ffd700 1 --2- 172.21.15.194:0/3753741021 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f572c02ff60 0x7f572c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:54:15.954 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:15.953+0000 7f5732ffd700 1 -- 172.21.15.194:0/3753741021 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5734040fb0 con 0x7f57400f6510 2024-01-18T06:54:15.954 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:15.953+0000 7f5745938700 1 --2- 172.21.15.194:0/3753741021 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f572c02ff60 0x7f572c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:54:15.954 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:15.953+0000 7f5745938700 1 --2- 172.21.15.194:0/3753741021 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f572c02ff60 0x7f572c032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f573c0096a0 tx=0x7f573c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:54:15.957 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:15.957+0000 7f5732ffd700 1 -- 172.21.15.194:0/3753741021 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f57340095f0 con 0x7f57400f6510 2024-01-18T06:54:16.164 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:16.165+0000 7f574839d700 1 -- 172.21.15.194:0/3753741021 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f57400fbf60 con 0x7f57400f6510 2024-01-18T06:54:16.165 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:16.166+0000 7f5732ffd700 1 -- 172.21.15.194:0/3753741021 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f573401db70 con 0x7f57400f6510 2024-01-18T06:54:16.165 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:54:16.165 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:54:16.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:16.167+0000 7f574839d700 1 -- 172.21.15.194:0/3753741021 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f572c02ff60 msgr2=0x7f572c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:16.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:16.167+0000 7f574839d700 1 --2- 172.21.15.194:0/3753741021 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f572c02ff60 0x7f572c032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f573c0096a0 tx=0x7f573c006b40).stop 2024-01-18T06:54:16.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:16.167+0000 7f574839d700 1 -- 172.21.15.194:0/3753741021 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57400f6510 msgr2=0x7f5740106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:16.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:16.167+0000 7f574839d700 1 --2- 172.21.15.194:0/3753741021 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57400f6510 0x7f5740106730 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f5734013510 tx=0x7f573400b040).stop 2024-01-18T06:54:16.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:16.167+0000 7f574839d700 1 -- 172.21.15.194:0/3753741021 shutdown_connections 2024-01-18T06:54:16.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:16.167+0000 7f574839d700 1 --2- 172.21.15.194:0/3753741021 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f572c02ff60 0x7f572c032420 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:16.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:16.167+0000 7f574839d700 1 --2- 172.21.15.194:0/3753741021 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57400f6510 0x7f5740106730 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:16.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:16.167+0000 7f574839d700 1 -- 172.21.15.194:0/3753741021 >> 172.21.15.194:0/3753741021 conn(0x7f57400f1c70 msgr2=0x7f57400f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:54:16.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:16.167+0000 7f574839d700 1 -- 172.21.15.194:0/3753741021 shutdown_connections 2024-01-18T06:54:16.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:16.167+0000 7f574839d700 1 -- 172.21.15.194:0/3753741021 wait complete. 2024-01-18T06:54:16.168 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:54:17.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:17 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:54:16.028131+0000 mgr.smithi047.fuhzzr (mgr.14162) 179 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:54:17.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:17 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:54:16.166761+0000 mon.smithi047 (mon.0) 248 : audit [DBG] from='client.? 172.21.15.194:0/3753741021' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:54:17.572 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:54:17.572 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:54:18.799 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:18.799+0000 7feb6d559700 1 -- 172.21.15.194:0/3297312537 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb680601e0 msgr2=0x7feb680f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:18.799 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:18.799+0000 7feb6d559700 1 --2- 172.21.15.194:0/3297312537 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb680601e0 0x7feb680f8f20 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7feb58009fb0 tx=0x7feb58009b30).stop 2024-01-18T06:54:18.799 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:18.799+0000 7feb6d559700 1 -- 172.21.15.194:0/3297312537 shutdown_connections 2024-01-18T06:54:18.799 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:18.799+0000 7feb6d559700 1 --2- 172.21.15.194:0/3297312537 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb680601e0 0x7feb680f8f20 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:18.799 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:18.799+0000 7feb6d559700 1 -- 172.21.15.194:0/3297312537 >> 172.21.15.194:0/3297312537 conn(0x7feb680f09a0 msgr2=0x7feb680f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:54:18.800 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:18.799+0000 7feb6d559700 1 -- 172.21.15.194:0/3297312537 shutdown_connections 2024-01-18T06:54:18.800 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:18.799+0000 7feb6d559700 1 -- 172.21.15.194:0/3297312537 wait complete. 2024-01-18T06:54:18.800 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:18.799+0000 7feb6d559700 1 Processor -- start 2024-01-18T06:54:18.800 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:18.799+0000 7feb6d559700 1 -- start start 2024-01-18T06:54:18.800 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:18.800+0000 7feb6d559700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb680601e0 0x7feb680f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:54:18.800 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:18.800+0000 7feb6d559700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb680f4e20 con 0x7feb680601e0 2024-01-18T06:54:18.800 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:18.800+0000 7feb67fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb680601e0 0x7feb680f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:54:18.800 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:18.800+0000 7feb67fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb680601e0 0x7feb680f69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:45660/0 (socket says 172.21.15.194:45660) 2024-01-18T06:54:18.800 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:18.800+0000 7feb67fff700 1 -- 172.21.15.194:0/1639087695 learned_addr learned my addr 172.21.15.194:0/1639087695 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:54:18.802 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:18.800+0000 7feb67fff700 1 -- 172.21.15.194:0/1639087695 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb58005040 con 0x7feb680601e0 2024-01-18T06:54:18.802 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:18.800+0000 7feb67fff700 1 --2- 172.21.15.194:0/1639087695 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb680601e0 0x7feb680f69a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7feb58013510 tx=0x7feb5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:54:18.802 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:18.800+0000 7feb64ff9700 1 -- 172.21.15.194:0/1639087695 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb58009320 con 0x7feb680601e0 2024-01-18T06:54:18.802 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:18.800+0000 7feb64ff9700 1 -- 172.21.15.194:0/1639087695 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feb58013f00 con 0x7feb680601e0 2024-01-18T06:54:18.802 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:18.800+0000 7feb6d559700 1 -- 172.21.15.194:0/1639087695 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb680f4fc0 con 0x7feb680601e0 2024-01-18T06:54:18.802 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:18.800+0000 7feb64ff9700 1 -- 172.21.15.194:0/1639087695 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb58014d90 con 0x7feb680601e0 2024-01-18T06:54:18.802 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:18.800+0000 7feb6d559700 1 -- 172.21.15.194:0/1639087695 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb680f5450 con 0x7feb680601e0 2024-01-18T06:54:18.802 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:18.801+0000 7feb64ff9700 1 -- 172.21.15.194:0/1639087695 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feb580095f0 con 0x7feb680601e0 2024-01-18T06:54:18.802 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:18.801+0000 7feb6d559700 1 -- 172.21.15.194:0/1639087695 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb68059100 con 0x7feb680601e0 2024-01-18T06:54:18.802 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:18.801+0000 7feb64ff9700 1 --2- 172.21.15.194:0/1639087695 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7feb5002ff60 0x7feb50032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:54:18.803 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:18.801+0000 7feb64ff9700 1 -- 172.21.15.194:0/1639087695 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feb58041000 con 0x7feb680601e0 2024-01-18T06:54:18.803 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:18.802+0000 7feb677fe700 1 --2- 172.21.15.194:0/1639087695 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7feb5002ff60 0x7feb50032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:54:18.803 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:18.802+0000 7feb677fe700 1 --2- 172.21.15.194:0/1639087695 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7feb5002ff60 0x7feb50032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7feb5c0096a0 tx=0x7feb5c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:54:18.805 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:18.806+0000 7feb64ff9700 1 -- 172.21.15.194:0/1639087695 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feb58014420 con 0x7feb680601e0 2024-01-18T06:54:19.009 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:19.009+0000 7feb6d559700 1 -- 172.21.15.194:0/1639087695 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feb680fbcd0 con 0x7feb680601e0 2024-01-18T06:54:19.009 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:19.010+0000 7feb64ff9700 1 -- 172.21.15.194:0/1639087695 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7feb5801d7f0 con 0x7feb680601e0 2024-01-18T06:54:19.010 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:54:19.010 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:54:19.011 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:19.012+0000 7feb6d559700 1 -- 172.21.15.194:0/1639087695 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7feb5002ff60 msgr2=0x7feb50032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:19.012 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:19.012+0000 7feb6d559700 1 --2- 172.21.15.194:0/1639087695 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7feb5002ff60 0x7feb50032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7feb5c0096a0 tx=0x7feb5c006b40).stop 2024-01-18T06:54:19.012 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:19.012+0000 7feb6d559700 1 -- 172.21.15.194:0/1639087695 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb680601e0 msgr2=0x7feb680f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:19.012 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:19.012+0000 7feb6d559700 1 --2- 172.21.15.194:0/1639087695 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb680601e0 0x7feb680f69a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7feb58013510 tx=0x7feb5800b040).stop 2024-01-18T06:54:19.012 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:19.012+0000 7feb6d559700 1 -- 172.21.15.194:0/1639087695 shutdown_connections 2024-01-18T06:54:19.012 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:19.012+0000 7feb6d559700 1 --2- 172.21.15.194:0/1639087695 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7feb5002ff60 0x7feb50032420 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:19.012 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:19.012+0000 7feb6d559700 1 --2- 172.21.15.194:0/1639087695 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb680601e0 0x7feb680f69a0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:19.012 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:19.012+0000 7feb6d559700 1 -- 172.21.15.194:0/1639087695 >> 172.21.15.194:0/1639087695 conn(0x7feb680f09a0 msgr2=0x7feb680f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:54:19.012 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:19.012+0000 7feb6d559700 1 -- 172.21.15.194:0/1639087695 shutdown_connections 2024-01-18T06:54:19.012 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:19.012+0000 7feb6d559700 1 -- 172.21.15.194:0/1639087695 wait complete. 2024-01-18T06:54:19.013 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:54:19.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:18 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:54:18.028479+0000 mgr.smithi047.fuhzzr (mgr.14162) 180 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:54:20.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:19 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:54:19.011125+0000 mon.smithi047 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.194:0/1639087695' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:54:20.398 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:54:20.398 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:54:21.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:20 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:54:20.028776+0000 mgr.smithi047.fuhzzr (mgr.14162) 181 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:54:21.653 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:21.653+0000 7f73d8740700 1 -- 172.21.15.194:0/1060408452 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73d0060280 msgr2=0x7f73d00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:21.654 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:21.653+0000 7f73d8740700 1 --2- 172.21.15.194:0/1060408452 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73d0060280 0x7f73d00f90f0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f73c0009fb0 tx=0x7f73c0009b30).stop 2024-01-18T06:54:21.654 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:21.653+0000 7f73d8740700 1 -- 172.21.15.194:0/1060408452 shutdown_connections 2024-01-18T06:54:21.654 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:21.653+0000 7f73d8740700 1 --2- 172.21.15.194:0/1060408452 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73d0060280 0x7f73d00f90f0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:21.654 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:21.653+0000 7f73d8740700 1 -- 172.21.15.194:0/1060408452 >> 172.21.15.194:0/1060408452 conn(0x7f73d00f0b70 msgr2=0x7f73d00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:54:21.654 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:21.654+0000 7f73d8740700 1 -- 172.21.15.194:0/1060408452 shutdown_connections 2024-01-18T06:54:21.654 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:21.654+0000 7f73d8740700 1 -- 172.21.15.194:0/1060408452 wait complete. 2024-01-18T06:54:21.654 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:21.654+0000 7f73d8740700 1 Processor -- start 2024-01-18T06:54:21.654 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:21.654+0000 7f73d8740700 1 -- start start 2024-01-18T06:54:21.654 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:21.654+0000 7f73d8740700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73d0060280 0x7f73d01066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:54:21.654 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:21.654+0000 7f73d8740700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d0106bd0 con 0x7f73d0060280 2024-01-18T06:54:21.656 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:21.655+0000 7f73d64dc700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73d0060280 0x7f73d01066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:54:21.656 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:21.655+0000 7f73d64dc700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73d0060280 0x7f73d01066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:45682/0 (socket says 172.21.15.194:45682) 2024-01-18T06:54:21.656 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:21.655+0000 7f73d64dc700 1 -- 172.21.15.194:0/3250868237 learned_addr learned my addr 172.21.15.194:0/3250868237 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:54:21.656 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:21.655+0000 7f73d64dc700 1 -- 172.21.15.194:0/3250868237 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c0005040 con 0x7f73d0060280 2024-01-18T06:54:21.656 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:21.655+0000 7f73d64dc700 1 --2- 172.21.15.194:0/3250868237 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73d0060280 0x7f73d01066e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f73c0013510 tx=0x7f73c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:54:21.656 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:21.655+0000 7f73c6ffd700 1 -- 172.21.15.194:0/3250868237 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f73c0009320 con 0x7f73d0060280 2024-01-18T06:54:21.657 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:21.655+0000 7f73d8740700 1 -- 172.21.15.194:0/3250868237 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73d0106dd0 con 0x7f73d0060280 2024-01-18T06:54:21.657 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:21.655+0000 7f73d8740700 1 -- 172.21.15.194:0/3250868237 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73d0107260 con 0x7f73d0060280 2024-01-18T06:54:21.657 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:21.655+0000 7f73c6ffd700 1 -- 172.21.15.194:0/3250868237 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f73c0013f00 con 0x7f73d0060280 2024-01-18T06:54:21.657 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:21.655+0000 7f73c6ffd700 1 -- 172.21.15.194:0/3250868237 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f73c0014d90 con 0x7f73d0060280 2024-01-18T06:54:21.657 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:21.656+0000 7f73c6ffd700 1 -- 172.21.15.194:0/3250868237 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f73c0014420 con 0x7f73d0060280 2024-01-18T06:54:21.657 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:21.656+0000 7f73d8740700 1 -- 172.21.15.194:0/3250868237 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73b8004fa0 con 0x7f73d0060280 2024-01-18T06:54:21.657 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:21.656+0000 7f73c6ffd700 1 --2- 172.21.15.194:0/3250868237 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f73bc02ffb0 0x7f73bc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:54:21.657 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:21.656+0000 7f73c6ffd700 1 -- 172.21.15.194:0/3250868237 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f73c00412d0 con 0x7f73d0060280 2024-01-18T06:54:21.657 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:21.656+0000 7f73d5cdb700 1 --2- 172.21.15.194:0/3250868237 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f73bc02ffb0 0x7f73bc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:54:21.657 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:21.657+0000 7f73d5cdb700 1 --2- 172.21.15.194:0/3250868237 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f73bc02ffb0 0x7f73bc032470 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f73cc0096a0 tx=0x7f73cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:54:21.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:21.661+0000 7f73c6ffd700 1 -- 172.21.15.194:0/3250868237 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f73c0012b20 con 0x7f73d0060280 2024-01-18T06:54:21.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:21.866+0000 7f73d8740700 1 -- 172.21.15.194:0/3250868237 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f73b8005670 con 0x7f73d0060280 2024-01-18T06:54:21.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:21.867+0000 7f73c6ffd700 1 -- 172.21.15.194:0/3250868237 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f73c0012b20 con 0x7f73d0060280 2024-01-18T06:54:21.867 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:54:21.867 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:54:21.868 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:21.869+0000 7f73d8740700 1 -- 172.21.15.194:0/3250868237 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f73bc02ffb0 msgr2=0x7f73bc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:21.868 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:21.869+0000 7f73d8740700 1 --2- 172.21.15.194:0/3250868237 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f73bc02ffb0 0x7f73bc032470 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f73cc0096a0 tx=0x7f73cc006b40).stop 2024-01-18T06:54:21.868 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:21.869+0000 7f73d8740700 1 -- 172.21.15.194:0/3250868237 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73d0060280 msgr2=0x7f73d01066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:21.868 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:21.869+0000 7f73d8740700 1 --2- 172.21.15.194:0/3250868237 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73d0060280 0x7f73d01066e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f73c0013510 tx=0x7f73c000b040).stop 2024-01-18T06:54:21.868 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:21.869+0000 7f73d8740700 1 -- 172.21.15.194:0/3250868237 shutdown_connections 2024-01-18T06:54:21.868 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:21.869+0000 7f73d8740700 1 --2- 172.21.15.194:0/3250868237 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f73bc02ffb0 0x7f73bc032470 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:21.868 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:21.869+0000 7f73d8740700 1 --2- 172.21.15.194:0/3250868237 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73d0060280 0x7f73d01066e0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:21.869 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:21.869+0000 7f73d8740700 1 -- 172.21.15.194:0/3250868237 >> 172.21.15.194:0/3250868237 conn(0x7f73d00f0b70 msgr2=0x7f73d00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:54:21.869 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:21.869+0000 7f73d8740700 1 -- 172.21.15.194:0/3250868237 shutdown_connections 2024-01-18T06:54:21.869 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:21.869+0000 7f73d8740700 1 -- 172.21.15.194:0/3250868237 wait complete. 2024-01-18T06:54:21.869 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:54:22.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:21 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:54:21.867990+0000 mon.smithi047 (mon.0) 250 : audit [DBG] from='client.? 172.21.15.194:0/3250868237' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:54:23.265 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:54:23.265 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:54:23.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:22 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:54:22.029108+0000 mgr.smithi047.fuhzzr (mgr.14162) 182 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:54:24.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:24.416+0000 7f5658381700 1 -- 172.21.15.194:0/2730555264 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f56500f6510 msgr2=0x7f56500f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:24.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:24.416+0000 7f5658381700 1 --2- 172.21.15.194:0/2730555264 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f56500f6510 0x7f56500f6930 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f564c009fb0 tx=0x7f564c009b30).stop 2024-01-18T06:54:24.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:24.416+0000 7f5658381700 1 -- 172.21.15.194:0/2730555264 shutdown_connections 2024-01-18T06:54:24.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:24.416+0000 7f5658381700 1 --2- 172.21.15.194:0/2730555264 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f56500f6510 0x7f56500f6930 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:24.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:24.416+0000 7f5658381700 1 -- 172.21.15.194:0/2730555264 >> 172.21.15.194:0/2730555264 conn(0x7f56500f1c70 msgr2=0x7f56500f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:54:24.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:24.416+0000 7f5658381700 1 -- 172.21.15.194:0/2730555264 shutdown_connections 2024-01-18T06:54:24.418 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:24.416+0000 7f5658381700 1 -- 172.21.15.194:0/2730555264 wait complete. 2024-01-18T06:54:24.418 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:24.416+0000 7f5658381700 1 Processor -- start 2024-01-18T06:54:24.418 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:24.416+0000 7f5658381700 1 -- start start 2024-01-18T06:54:24.418 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:24.416+0000 7f5658381700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f56500f6510 0x7f5650106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:54:24.418 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:24.416+0000 7f5658381700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5650106c20 con 0x7f56500f6510 2024-01-18T06:54:24.418 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:24.417+0000 7f565611d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f56500f6510 0x7f5650106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:54:24.418 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:24.417+0000 7f565611d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f56500f6510 0x7f5650106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:45698/0 (socket says 172.21.15.194:45698) 2024-01-18T06:54:24.418 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:24.417+0000 7f565611d700 1 -- 172.21.15.194:0/1486024725 learned_addr learned my addr 172.21.15.194:0/1486024725 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:54:24.418 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:24.417+0000 7f565611d700 1 -- 172.21.15.194:0/1486024725 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f564c005040 con 0x7f56500f6510 2024-01-18T06:54:24.418 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:24.417+0000 7f565611d700 1 --2- 172.21.15.194:0/1486024725 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f56500f6510 0x7f5650106730 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f564c013510 tx=0x7f564c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:54:24.418 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:24.417+0000 7f5646ffd700 1 -- 172.21.15.194:0/1486024725 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f564c009320 con 0x7f56500f6510 2024-01-18T06:54:24.418 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:24.417+0000 7f5646ffd700 1 -- 172.21.15.194:0/1486024725 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f564c013f00 con 0x7f56500f6510 2024-01-18T06:54:24.418 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:24.417+0000 7f5658381700 1 -- 172.21.15.194:0/1486024725 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5650106e20 con 0x7f56500f6510 2024-01-18T06:54:24.418 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:24.418+0000 7f5646ffd700 1 -- 172.21.15.194:0/1486024725 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f564c014d90 con 0x7f56500f6510 2024-01-18T06:54:24.419 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:24.418+0000 7f5658381700 1 -- 172.21.15.194:0/1486024725 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56501072b0 con 0x7f56500f6510 2024-01-18T06:54:24.419 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:24.418+0000 7f5658381700 1 -- 172.21.15.194:0/1486024725 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5650045ad0 con 0x7f56500f6510 2024-01-18T06:54:24.420 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:24.418+0000 7f5646ffd700 1 -- 172.21.15.194:0/1486024725 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f564c014420 con 0x7f56500f6510 2024-01-18T06:54:24.420 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:24.419+0000 7f5646ffd700 1 --2- 172.21.15.194:0/1486024725 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f563c02ff60 0x7f563c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:54:24.420 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:24.419+0000 7f5646ffd700 1 -- 172.21.15.194:0/1486024725 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f564c041000 con 0x7f56500f6510 2024-01-18T06:54:24.420 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:24.419+0000 7f565591c700 1 --2- 172.21.15.194:0/1486024725 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f563c02ff60 0x7f563c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:54:24.420 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:24.419+0000 7f565591c700 1 --2- 172.21.15.194:0/1486024725 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f563c02ff60 0x7f563c032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f56400096a0 tx=0x7f5640006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:54:24.423 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:24.424+0000 7f5646ffd700 1 -- 172.21.15.194:0/1486024725 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f564c0095f0 con 0x7f56500f6510 2024-01-18T06:54:24.627 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:24.628+0000 7f5658381700 1 -- 172.21.15.194:0/1486024725 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f56500fbf60 con 0x7f56500f6510 2024-01-18T06:54:24.628 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:24.628+0000 7f5646ffd700 1 -- 172.21.15.194:0/1486024725 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f564c01db70 con 0x7f56500f6510 2024-01-18T06:54:24.628 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:54:24.628 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:54:24.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:24.630+0000 7f5658381700 1 -- 172.21.15.194:0/1486024725 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f563c02ff60 msgr2=0x7f563c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:24.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:24.630+0000 7f5658381700 1 --2- 172.21.15.194:0/1486024725 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f563c02ff60 0x7f563c032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f56400096a0 tx=0x7f5640006b40).stop 2024-01-18T06:54:24.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:24.630+0000 7f5658381700 1 -- 172.21.15.194:0/1486024725 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f56500f6510 msgr2=0x7f5650106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:24.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:24.630+0000 7f5658381700 1 --2- 172.21.15.194:0/1486024725 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f56500f6510 0x7f5650106730 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f564c013510 tx=0x7f564c00b040).stop 2024-01-18T06:54:24.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:24.630+0000 7f5658381700 1 -- 172.21.15.194:0/1486024725 shutdown_connections 2024-01-18T06:54:24.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:24.630+0000 7f5658381700 1 --2- 172.21.15.194:0/1486024725 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f563c02ff60 0x7f563c032420 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:24.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:24.630+0000 7f5658381700 1 --2- 172.21.15.194:0/1486024725 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f56500f6510 0x7f5650106730 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:24.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:24.630+0000 7f5658381700 1 -- 172.21.15.194:0/1486024725 >> 172.21.15.194:0/1486024725 conn(0x7f56500f1c70 msgr2=0x7f56500f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:54:24.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:24.630+0000 7f5658381700 1 -- 172.21.15.194:0/1486024725 shutdown_connections 2024-01-18T06:54:24.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:24.630+0000 7f5658381700 1 -- 172.21.15.194:0/1486024725 wait complete. 2024-01-18T06:54:24.631 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:54:25.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:25 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:54:24.029406+0000 mgr.smithi047.fuhzzr (mgr.14162) 183 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:54:25.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:25 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:54:24.629351+0000 mon.smithi047 (mon.0) 251 : audit [DBG] from='client.? 172.21.15.194:0/1486024725' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:54:26.002 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:54:26.003 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:54:27.247 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:27.247+0000 7fcb90481700 1 -- 172.21.15.194:0/1143439539 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcb880f5410 msgr2=0x7fcb880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:27.247 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:27.247+0000 7fcb90481700 1 --2- 172.21.15.194:0/1143439539 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcb880f5410 0x7fcb880f5830 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7fcb78009fb0 tx=0x7fcb78009b30).stop 2024-01-18T06:54:27.247 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:27.247+0000 7fcb90481700 1 -- 172.21.15.194:0/1143439539 shutdown_connections 2024-01-18T06:54:27.247 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:27.247+0000 7fcb90481700 1 --2- 172.21.15.194:0/1143439539 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcb880f5410 0x7fcb880f5830 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:27.247 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:27.247+0000 7fcb90481700 1 -- 172.21.15.194:0/1143439539 >> 172.21.15.194:0/1143439539 conn(0x7fcb880f0b70 msgr2=0x7fcb880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:54:27.247 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:27.247+0000 7fcb90481700 1 -- 172.21.15.194:0/1143439539 shutdown_connections 2024-01-18T06:54:27.248 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:27.247+0000 7fcb90481700 1 -- 172.21.15.194:0/1143439539 wait complete. 2024-01-18T06:54:27.249 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:27.248+0000 7fcb90481700 1 Processor -- start 2024-01-18T06:54:27.249 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:27.248+0000 7fcb90481700 1 -- start start 2024-01-18T06:54:27.250 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:27.248+0000 7fcb90481700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcb880f5410 0x7fcb88109390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:54:27.250 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:27.248+0000 7fcb90481700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb88060280 con 0x7fcb880f5410 2024-01-18T06:54:27.250 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:27.249+0000 7fcb8e21d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcb880f5410 0x7fcb88109390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:54:27.250 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:27.249+0000 7fcb8e21d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcb880f5410 0x7fcb88109390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:45730/0 (socket says 172.21.15.194:45730) 2024-01-18T06:54:27.250 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:27.249+0000 7fcb8e21d700 1 -- 172.21.15.194:0/1547438152 learned_addr learned my addr 172.21.15.194:0/1547438152 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:54:27.250 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:27.249+0000 7fcb8e21d700 1 -- 172.21.15.194:0/1547438152 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb78005040 con 0x7fcb880f5410 2024-01-18T06:54:27.250 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:27.249+0000 7fcb8e21d700 1 --2- 172.21.15.194:0/1547438152 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcb880f5410 0x7fcb88109390 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7fcb78013510 tx=0x7fcb7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:54:27.250 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:27.249+0000 7fcb7effd700 1 -- 172.21.15.194:0/1547438152 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb78009320 con 0x7fcb880f5410 2024-01-18T06:54:27.250 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:27.249+0000 7fcb90481700 1 -- 172.21.15.194:0/1547438152 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb88060480 con 0x7fcb880f5410 2024-01-18T06:54:27.250 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:27.249+0000 7fcb7effd700 1 -- 172.21.15.194:0/1547438152 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcb78013f00 con 0x7fcb880f5410 2024-01-18T06:54:27.250 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:27.249+0000 7fcb7effd700 1 -- 172.21.15.194:0/1547438152 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb78014d90 con 0x7fcb880f5410 2024-01-18T06:54:27.250 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:27.249+0000 7fcb90481700 1 -- 172.21.15.194:0/1547438152 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb88109ac0 con 0x7fcb880f5410 2024-01-18T06:54:27.250 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:27.250+0000 7fcb7effd700 1 -- 172.21.15.194:0/1547438152 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcb780095f0 con 0x7fcb880f5410 2024-01-18T06:54:27.250 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:27.250+0000 7fcb90481700 1 -- 172.21.15.194:0/1547438152 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb88045ad0 con 0x7fcb880f5410 2024-01-18T06:54:27.251 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:27.250+0000 7fcb7effd700 1 --2- 172.21.15.194:0/1547438152 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fcb7402ff60 0x7fcb74032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:54:27.251 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:27.250+0000 7fcb7effd700 1 -- 172.21.15.194:0/1547438152 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcb78041000 con 0x7fcb880f5410 2024-01-18T06:54:27.251 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:27.251+0000 7fcb8da1c700 1 --2- 172.21.15.194:0/1547438152 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fcb7402ff60 0x7fcb74032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:54:27.251 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:27.251+0000 7fcb8da1c700 1 --2- 172.21.15.194:0/1547438152 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fcb7402ff60 0x7fcb74032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fcb840096a0 tx=0x7fcb84006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:54:27.254 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:27.255+0000 7fcb7effd700 1 -- 172.21.15.194:0/1547438152 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcb78014420 con 0x7fcb880f5410 2024-01-18T06:54:27.459 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:27.459+0000 7fcb90481700 1 -- 172.21.15.194:0/1547438152 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcb880fae60 con 0x7fcb880f5410 2024-01-18T06:54:27.459 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:27.460+0000 7fcb7effd700 1 -- 172.21.15.194:0/1547438152 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcb78010230 con 0x7fcb880f5410 2024-01-18T06:54:27.460 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:54:27.460 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:54:27.461 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:27.461+0000 7fcb90481700 1 -- 172.21.15.194:0/1547438152 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fcb7402ff60 msgr2=0x7fcb74032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:27.461 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:27.461+0000 7fcb90481700 1 --2- 172.21.15.194:0/1547438152 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fcb7402ff60 0x7fcb74032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fcb840096a0 tx=0x7fcb84006b40).stop 2024-01-18T06:54:27.461 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:27.461+0000 7fcb90481700 1 -- 172.21.15.194:0/1547438152 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcb880f5410 msgr2=0x7fcb88109390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:27.461 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:27.461+0000 7fcb90481700 1 --2- 172.21.15.194:0/1547438152 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcb880f5410 0x7fcb88109390 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7fcb78013510 tx=0x7fcb7800b040).stop 2024-01-18T06:54:27.461 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:27.461+0000 7fcb90481700 1 -- 172.21.15.194:0/1547438152 shutdown_connections 2024-01-18T06:54:27.461 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:27.461+0000 7fcb90481700 1 --2- 172.21.15.194:0/1547438152 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fcb7402ff60 0x7fcb74032420 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:27.461 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:27.461+0000 7fcb90481700 1 --2- 172.21.15.194:0/1547438152 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcb880f5410 0x7fcb88109390 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:27.462 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:27.461+0000 7fcb90481700 1 -- 172.21.15.194:0/1547438152 >> 172.21.15.194:0/1547438152 conn(0x7fcb880f0b70 msgr2=0x7fcb880f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:54:27.462 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:27.461+0000 7fcb90481700 1 -- 172.21.15.194:0/1547438152 shutdown_connections 2024-01-18T06:54:27.462 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:27.461+0000 7fcb90481700 1 -- 172.21.15.194:0/1547438152 wait complete. 2024-01-18T06:54:27.462 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:54:27.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:27 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:54:26.029747+0000 mgr.smithi047.fuhzzr (mgr.14162) 184 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:54:28.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:28 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:54:27.460884+0000 mon.smithi047 (mon.0) 252 : audit [DBG] from='client.? 172.21.15.194:0/1547438152' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:54:28.834 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:54:28.834 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:54:29.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:29 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:54:28.030023+0000 mgr.smithi047.fuhzzr (mgr.14162) 185 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:54:30.088 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:30.086+0000 7f1c7a573700 1 -- 172.21.15.194:0/2786009649 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c740f5410 msgr2=0x7f1c740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:30.088 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:30.086+0000 7f1c7a573700 1 --2- 172.21.15.194:0/2786009649 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c740f5410 0x7f1c740f5830 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f1c64009fb0 tx=0x7f1c64009b30).stop 2024-01-18T06:54:30.088 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:30.087+0000 7f1c7a573700 1 -- 172.21.15.194:0/2786009649 shutdown_connections 2024-01-18T06:54:30.088 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:30.087+0000 7f1c7a573700 1 --2- 172.21.15.194:0/2786009649 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c740f5410 0x7f1c740f5830 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:30.089 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:30.087+0000 7f1c7a573700 1 -- 172.21.15.194:0/2786009649 >> 172.21.15.194:0/2786009649 conn(0x7f1c740f0b70 msgr2=0x7f1c740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:54:30.089 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:30.087+0000 7f1c7a573700 1 -- 172.21.15.194:0/2786009649 shutdown_connections 2024-01-18T06:54:30.089 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:30.087+0000 7f1c7a573700 1 -- 172.21.15.194:0/2786009649 wait complete. 2024-01-18T06:54:30.089 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:30.087+0000 7f1c7a573700 1 Processor -- start 2024-01-18T06:54:30.089 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:30.087+0000 7f1c7a573700 1 -- start start 2024-01-18T06:54:30.089 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:30.087+0000 7f1c7a573700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c740f5410 0x7f1c74109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:54:30.089 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:30.087+0000 7f1c7a573700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c74060220 con 0x7f1c740f5410 2024-01-18T06:54:30.089 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:30.088+0000 7f1c73fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c740f5410 0x7f1c74109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:54:30.089 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:30.088+0000 7f1c73fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c740f5410 0x7f1c74109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:46866/0 (socket says 172.21.15.194:46866) 2024-01-18T06:54:30.089 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:30.088+0000 7f1c73fff700 1 -- 172.21.15.194:0/2013232631 learned_addr learned my addr 172.21.15.194:0/2013232631 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:54:30.089 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:30.088+0000 7f1c73fff700 1 -- 172.21.15.194:0/2013232631 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c64005040 con 0x7f1c740f5410 2024-01-18T06:54:30.089 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:30.088+0000 7f1c73fff700 1 --2- 172.21.15.194:0/2013232631 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c740f5410 0x7f1c74109450 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f1c64014000 tx=0x7f1c6400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:54:30.089 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:30.088+0000 7f1c70ff9700 1 -- 172.21.15.194:0/2013232631 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c640095f0 con 0x7f1c740f5410 2024-01-18T06:54:30.090 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:30.088+0000 7f1c7a573700 1 -- 172.21.15.194:0/2013232631 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c74060420 con 0x7f1c740f5410 2024-01-18T06:54:30.090 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:30.088+0000 7f1c70ff9700 1 -- 172.21.15.194:0/2013232631 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1c64007e70 con 0x7f1c740f5410 2024-01-18T06:54:30.090 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:30.088+0000 7f1c70ff9700 1 -- 172.21.15.194:0/2013232631 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c64014ea0 con 0x7f1c740f5410 2024-01-18T06:54:30.090 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:30.088+0000 7f1c7a573700 1 -- 172.21.15.194:0/2013232631 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c74109b00 con 0x7f1c740f5410 2024-01-18T06:54:30.091 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:30.089+0000 7f1c7a573700 1 -- 172.21.15.194:0/2013232631 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c74059100 con 0x7f1c740f5410 2024-01-18T06:54:30.091 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:30.089+0000 7f1c70ff9700 1 -- 172.21.15.194:0/2013232631 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1c64025070 con 0x7f1c740f5410 2024-01-18T06:54:30.091 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:30.089+0000 7f1c70ff9700 1 --2- 172.21.15.194:0/2013232631 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f1c5c02ff60 0x7f1c5c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:54:30.091 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:30.089+0000 7f1c70ff9700 1 -- 172.21.15.194:0/2013232631 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1c64040a10 con 0x7f1c740f5410 2024-01-18T06:54:30.091 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:30.090+0000 7f1c737fe700 1 --2- 172.21.15.194:0/2013232631 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f1c5c02ff60 0x7f1c5c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:54:30.091 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:30.090+0000 7f1c737fe700 1 --2- 172.21.15.194:0/2013232631 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f1c5c02ff60 0x7f1c5c032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f1c680096a0 tx=0x7f1c68006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:54:30.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:30.094+0000 7f1c70ff9700 1 -- 172.21.15.194:0/2013232631 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1c640079c0 con 0x7f1c740f5410 2024-01-18T06:54:30.297 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:30.297+0000 7f1c7a573700 1 -- 172.21.15.194:0/2013232631 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1c74045ad0 con 0x7f1c740f5410 2024-01-18T06:54:30.297 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:30.298+0000 7f1c70ff9700 1 -- 172.21.15.194:0/2013232631 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1c6400f300 con 0x7f1c740f5410 2024-01-18T06:54:30.298 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:54:30.298 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:54:30.299 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:30.299+0000 7f1c7a573700 1 -- 172.21.15.194:0/2013232631 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f1c5c02ff60 msgr2=0x7f1c5c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:30.299 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:30.299+0000 7f1c7a573700 1 --2- 172.21.15.194:0/2013232631 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f1c5c02ff60 0x7f1c5c032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f1c680096a0 tx=0x7f1c68006b40).stop 2024-01-18T06:54:30.299 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:30.299+0000 7f1c7a573700 1 -- 172.21.15.194:0/2013232631 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c740f5410 msgr2=0x7f1c74109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:30.299 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:30.299+0000 7f1c7a573700 1 --2- 172.21.15.194:0/2013232631 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c740f5410 0x7f1c74109450 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f1c64014000 tx=0x7f1c6400b040).stop 2024-01-18T06:54:30.299 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:30.300+0000 7f1c7a573700 1 -- 172.21.15.194:0/2013232631 shutdown_connections 2024-01-18T06:54:30.299 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:30.300+0000 7f1c7a573700 1 --2- 172.21.15.194:0/2013232631 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f1c5c02ff60 0x7f1c5c032420 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:30.299 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:30.300+0000 7f1c7a573700 1 --2- 172.21.15.194:0/2013232631 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c740f5410 0x7f1c74109450 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:30.299 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:30.300+0000 7f1c7a573700 1 -- 172.21.15.194:0/2013232631 >> 172.21.15.194:0/2013232631 conn(0x7f1c740f0b70 msgr2=0x7f1c740f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:54:30.299 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:30.300+0000 7f1c7a573700 1 -- 172.21.15.194:0/2013232631 shutdown_connections 2024-01-18T06:54:30.300 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:30.300+0000 7f1c7a573700 1 -- 172.21.15.194:0/2013232631 wait complete. 2024-01-18T06:54:30.300 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:54:31.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:31 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:54:30.030381+0000 mgr.smithi047.fuhzzr (mgr.14162) 186 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:54:31.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:31 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:54:30.117800+0000 mon.smithi047 (mon.0) 253 : audit [INF] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.fuhzzr/mirror_snapshot_schedule"}]: dispatch 2024-01-18T06:54:31.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:31 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:54:30.118172+0000 mon.smithi047 (mon.0) 254 : audit [INF] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.fuhzzr/trash_purge_schedule"}]: dispatch 2024-01-18T06:54:31.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:31 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:54:30.299098+0000 mon.smithi047 (mon.0) 255 : audit [DBG] from='client.? 172.21.15.194:0/2013232631' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:54:31.678 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:54:31.678 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:54:32.871 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:32.869+0000 7fe3493cf700 1 -- 172.21.15.194:0/2269214040 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe3440f51a0 msgr2=0x7fe3440f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:32.871 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:32.869+0000 7fe3493cf700 1 --2- 172.21.15.194:0/2269214040 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe3440f51a0 0x7fe3440f55c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7fe334009fb0 tx=0x7fe334009b30).stop 2024-01-18T06:54:32.871 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:32.869+0000 7fe3493cf700 1 -- 172.21.15.194:0/2269214040 shutdown_connections 2024-01-18T06:54:32.871 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:32.869+0000 7fe3493cf700 1 --2- 172.21.15.194:0/2269214040 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe3440f51a0 0x7fe3440f55c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:32.871 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:32.869+0000 7fe3493cf700 1 -- 172.21.15.194:0/2269214040 >> 172.21.15.194:0/2269214040 conn(0x7fe3440f09a0 msgr2=0x7fe3440f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:54:32.871 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:32.869+0000 7fe3493cf700 1 -- 172.21.15.194:0/2269214040 shutdown_connections 2024-01-18T06:54:32.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:32.869+0000 7fe3493cf700 1 -- 172.21.15.194:0/2269214040 wait complete. 2024-01-18T06:54:32.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:32.870+0000 7fe3493cf700 1 Processor -- start 2024-01-18T06:54:32.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:32.870+0000 7fe3493cf700 1 -- start start 2024-01-18T06:54:32.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:32.870+0000 7fe3493cf700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe3440f51a0 0x7fe3441064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:54:32.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:32.870+0000 7fe3493cf700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3441069a0 con 0x7fe3440f51a0 2024-01-18T06:54:32.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:32.871+0000 7fe343fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe3440f51a0 0x7fe3441064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:54:32.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:32.871+0000 7fe343fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe3440f51a0 0x7fe3441064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:46898/0 (socket says 172.21.15.194:46898) 2024-01-18T06:54:32.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:32.871+0000 7fe343fff700 1 -- 172.21.15.194:0/2674086392 learned_addr learned my addr 172.21.15.194:0/2674086392 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:54:32.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:32.871+0000 7fe343fff700 1 -- 172.21.15.194:0/2674086392 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe334005040 con 0x7fe3440f51a0 2024-01-18T06:54:32.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:32.871+0000 7fe343fff700 1 --2- 172.21.15.194:0/2674086392 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe3440f51a0 0x7fe3441064b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7fe334013510 tx=0x7fe33400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:54:32.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:32.871+0000 7fe340ff9700 1 -- 172.21.15.194:0/2674086392 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe334009320 con 0x7fe3440f51a0 2024-01-18T06:54:32.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:32.871+0000 7fe3493cf700 1 -- 172.21.15.194:0/2674086392 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe344106ba0 con 0x7fe3440f51a0 2024-01-18T06:54:32.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:32.871+0000 7fe340ff9700 1 -- 172.21.15.194:0/2674086392 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe334013f00 con 0x7fe3440f51a0 2024-01-18T06:54:32.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:32.871+0000 7fe340ff9700 1 -- 172.21.15.194:0/2674086392 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe334014dc0 con 0x7fe3440f51a0 2024-01-18T06:54:32.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:32.871+0000 7fe3493cf700 1 -- 172.21.15.194:0/2674086392 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe344107030 con 0x7fe3440f51a0 2024-01-18T06:54:32.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:32.872+0000 7fe340ff9700 1 -- 172.21.15.194:0/2674086392 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe3340263a0 con 0x7fe3440f51a0 2024-01-18T06:54:32.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:32.872+0000 7fe3493cf700 1 -- 172.21.15.194:0/2674086392 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe344045ad0 con 0x7fe3440f51a0 2024-01-18T06:54:32.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:32.872+0000 7fe340ff9700 1 --2- 172.21.15.194:0/2674086392 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fe32c02ffb0 0x7fe32c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:54:32.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:32.872+0000 7fe340ff9700 1 -- 172.21.15.194:0/2674086392 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe334041120 con 0x7fe3440f51a0 2024-01-18T06:54:32.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:32.873+0000 7fe3437fe700 1 --2- 172.21.15.194:0/2674086392 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fe32c02ffb0 0x7fe32c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:54:32.875 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:32.873+0000 7fe3437fe700 1 --2- 172.21.15.194:0/2674086392 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fe32c02ffb0 0x7fe32c032470 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fe3380096a0 tx=0x7fe338006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:54:32.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:32.877+0000 7fe340ff9700 1 -- 172.21.15.194:0/2674086392 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe3340126f0 con 0x7fe3440f51a0 2024-01-18T06:54:33.084 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:33.083+0000 7fe3493cf700 1 -- 172.21.15.194:0/2674086392 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe344107ee0 con 0x7fe3440f51a0 2024-01-18T06:54:33.084 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:33.084+0000 7fe340ff9700 1 -- 172.21.15.194:0/2674086392 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe33400f340 con 0x7fe3440f51a0 2024-01-18T06:54:33.084 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:54:33.084 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:54:33.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:33.086+0000 7fe3493cf700 1 -- 172.21.15.194:0/2674086392 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fe32c02ffb0 msgr2=0x7fe32c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:33.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:33.086+0000 7fe3493cf700 1 --2- 172.21.15.194:0/2674086392 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fe32c02ffb0 0x7fe32c032470 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fe3380096a0 tx=0x7fe338006b40).stop 2024-01-18T06:54:33.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:33.086+0000 7fe3493cf700 1 -- 172.21.15.194:0/2674086392 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe3440f51a0 msgr2=0x7fe3441064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:33.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:33.086+0000 7fe3493cf700 1 --2- 172.21.15.194:0/2674086392 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe3440f51a0 0x7fe3441064b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7fe334013510 tx=0x7fe33400b040).stop 2024-01-18T06:54:33.086 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:33.086+0000 7fe3493cf700 1 -- 172.21.15.194:0/2674086392 shutdown_connections 2024-01-18T06:54:33.086 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:33.086+0000 7fe3493cf700 1 --2- 172.21.15.194:0/2674086392 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fe32c02ffb0 0x7fe32c032470 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:33.086 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:33.086+0000 7fe3493cf700 1 --2- 172.21.15.194:0/2674086392 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe3440f51a0 0x7fe3441064b0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:33.086 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:33.086+0000 7fe3493cf700 1 -- 172.21.15.194:0/2674086392 >> 172.21.15.194:0/2674086392 conn(0x7fe3440f09a0 msgr2=0x7fe3440f1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:54:33.086 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:33.086+0000 7fe3493cf700 1 -- 172.21.15.194:0/2674086392 shutdown_connections 2024-01-18T06:54:33.086 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:33.086+0000 7fe3493cf700 1 -- 172.21.15.194:0/2674086392 wait complete. 2024-01-18T06:54:33.086 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:54:33.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:33 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:54:32.030697+0000 mgr.smithi047.fuhzzr (mgr.14162) 187 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:54:33.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:33 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:54:33.084973+0000 mon.smithi047 (mon.0) 256 : audit [DBG] from='client.? 172.21.15.194:0/2674086392' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:54:34.475 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:54:34.475 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:54:35.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:35 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:54:34.031058+0000 mgr.smithi047.fuhzzr (mgr.14162) 188 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:54:35.653 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:35.652+0000 7fa64af5a700 1 -- 172.21.15.194:0/3685835298 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa6440f62a0 msgr2=0x7fa6440f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:35.653 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:35.652+0000 7fa64af5a700 1 --2- 172.21.15.194:0/3685835298 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa6440f62a0 0x7fa6440f66c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7fa640009fb0 tx=0x7fa640009b30).stop 2024-01-18T06:54:35.653 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:35.652+0000 7fa64af5a700 1 -- 172.21.15.194:0/3685835298 shutdown_connections 2024-01-18T06:54:35.653 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:35.652+0000 7fa64af5a700 1 --2- 172.21.15.194:0/3685835298 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa6440f62a0 0x7fa6440f66c0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:35.653 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:35.652+0000 7fa64af5a700 1 -- 172.21.15.194:0/3685835298 >> 172.21.15.194:0/3685835298 conn(0x7fa6440f1aa0 msgr2=0x7fa6440f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:54:35.653 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:35.652+0000 7fa64af5a700 1 -- 172.21.15.194:0/3685835298 shutdown_connections 2024-01-18T06:54:35.653 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:35.652+0000 7fa64af5a700 1 -- 172.21.15.194:0/3685835298 wait complete. 2024-01-18T06:54:35.654 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:35.652+0000 7fa64af5a700 1 Processor -- start 2024-01-18T06:54:35.654 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:35.652+0000 7fa64af5a700 1 -- start start 2024-01-18T06:54:35.654 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:35.653+0000 7fa64af5a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa6440f62a0 0x7fa6441064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:54:35.654 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:35.653+0000 7fa64af5a700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6441069a0 con 0x7fa6440f62a0 2024-01-18T06:54:35.654 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:35.653+0000 7fa649f58700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa6440f62a0 0x7fa6441064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:54:35.654 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:35.653+0000 7fa649f58700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa6440f62a0 0x7fa6441064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:46916/0 (socket says 172.21.15.194:46916) 2024-01-18T06:54:35.654 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:35.653+0000 7fa649f58700 1 -- 172.21.15.194:0/745312314 learned_addr learned my addr 172.21.15.194:0/745312314 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:54:35.654 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:35.653+0000 7fa649f58700 1 -- 172.21.15.194:0/745312314 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa640005040 con 0x7fa6440f62a0 2024-01-18T06:54:35.654 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:35.653+0000 7fa649f58700 1 --2- 172.21.15.194:0/745312314 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa6440f62a0 0x7fa6441064b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fa640013510 tx=0x7fa64000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:54:35.654 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:35.653+0000 7fa63a7fc700 1 -- 172.21.15.194:0/745312314 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa640009320 con 0x7fa6440f62a0 2024-01-18T06:54:35.654 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:35.653+0000 7fa64af5a700 1 -- 172.21.15.194:0/745312314 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa644106ba0 con 0x7fa6440f62a0 2024-01-18T06:54:35.654 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:35.653+0000 7fa63a7fc700 1 -- 172.21.15.194:0/745312314 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa640013f00 con 0x7fa6440f62a0 2024-01-18T06:54:35.654 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:35.653+0000 7fa63a7fc700 1 -- 172.21.15.194:0/745312314 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa640014dc0 con 0x7fa6440f62a0 2024-01-18T06:54:35.654 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:35.653+0000 7fa64af5a700 1 -- 172.21.15.194:0/745312314 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa644107030 con 0x7fa6440f62a0 2024-01-18T06:54:35.655 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:35.654+0000 7fa63a7fc700 1 -- 172.21.15.194:0/745312314 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa640014420 con 0x7fa6440f62a0 2024-01-18T06:54:35.655 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:35.654+0000 7fa64af5a700 1 -- 172.21.15.194:0/745312314 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa644045ad0 con 0x7fa6440f62a0 2024-01-18T06:54:35.655 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:35.654+0000 7fa63a7fc700 1 --2- 172.21.15.194:0/745312314 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa63002ff60 0x7fa630032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:54:35.655 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:35.654+0000 7fa63a7fc700 1 -- 172.21.15.194:0/745312314 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa6400410f0 con 0x7fa6440f62a0 2024-01-18T06:54:35.656 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:35.655+0000 7fa649757700 1 --2- 172.21.15.194:0/745312314 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa63002ff60 0x7fa630032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:54:35.656 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:35.655+0000 7fa649757700 1 --2- 172.21.15.194:0/745312314 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa63002ff60 0x7fa630032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fa6340096a0 tx=0x7fa634006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:54:35.658 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:35.659+0000 7fa63a7fc700 1 -- 172.21.15.194:0/745312314 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa6400095f0 con 0x7fa6440f62a0 2024-01-18T06:54:35.863 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:35.864+0000 7fa64af5a700 1 -- 172.21.15.194:0/745312314 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa644107ee0 con 0x7fa6440f62a0 2024-01-18T06:54:35.864 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:35.865+0000 7fa63a7fc700 1 -- 172.21.15.194:0/745312314 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa64001cb10 con 0x7fa6440f62a0 2024-01-18T06:54:35.864 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:54:35.864 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:54:35.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:35.866+0000 7fa64af5a700 1 -- 172.21.15.194:0/745312314 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa63002ff60 msgr2=0x7fa630032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:35.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:35.866+0000 7fa64af5a700 1 --2- 172.21.15.194:0/745312314 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa63002ff60 0x7fa630032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fa6340096a0 tx=0x7fa634006b40).stop 2024-01-18T06:54:35.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:35.866+0000 7fa64af5a700 1 -- 172.21.15.194:0/745312314 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa6440f62a0 msgr2=0x7fa6441064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:35.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:35.866+0000 7fa64af5a700 1 --2- 172.21.15.194:0/745312314 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa6440f62a0 0x7fa6441064b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fa640013510 tx=0x7fa64000b040).stop 2024-01-18T06:54:35.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:35.866+0000 7fa64af5a700 1 -- 172.21.15.194:0/745312314 shutdown_connections 2024-01-18T06:54:35.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:35.866+0000 7fa64af5a700 1 --2- 172.21.15.194:0/745312314 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa63002ff60 0x7fa630032420 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:35.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:35.866+0000 7fa64af5a700 1 --2- 172.21.15.194:0/745312314 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa6440f62a0 0x7fa6441064b0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:35.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:35.866+0000 7fa64af5a700 1 -- 172.21.15.194:0/745312314 >> 172.21.15.194:0/745312314 conn(0x7fa6440f1aa0 msgr2=0x7fa6440f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:54:35.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:35.866+0000 7fa64af5a700 1 -- 172.21.15.194:0/745312314 shutdown_connections 2024-01-18T06:54:35.867 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:35.866+0000 7fa64af5a700 1 -- 172.21.15.194:0/745312314 wait complete. 2024-01-18T06:54:35.867 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:54:36.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:36 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:54:35.865791+0000 mon.smithi047 (mon.0) 257 : audit [DBG] from='client.? 172.21.15.194:0/745312314' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:54:37.256 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:54:37.256 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:54:37.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:54:36.031443+0000 mgr.smithi047.fuhzzr (mgr.14162) 189 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:54:38.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:38.412+0000 7f4649f12700 1 -- 172.21.15.194:0/2241781306 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46440f5410 msgr2=0x7f46440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:38.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:38.412+0000 7f4649f12700 1 --2- 172.21.15.194:0/2241781306 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46440f5410 0x7f46440f5830 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f4638009fb0 tx=0x7f4638009b30).stop 2024-01-18T06:54:38.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:38.412+0000 7f4649f12700 1 -- 172.21.15.194:0/2241781306 shutdown_connections 2024-01-18T06:54:38.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:38.412+0000 7f4649f12700 1 --2- 172.21.15.194:0/2241781306 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46440f5410 0x7f46440f5830 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:38.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:38.412+0000 7f4649f12700 1 -- 172.21.15.194:0/2241781306 >> 172.21.15.194:0/2241781306 conn(0x7f46440f0b70 msgr2=0x7f46440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:54:38.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:38.412+0000 7f4649f12700 1 -- 172.21.15.194:0/2241781306 shutdown_connections 2024-01-18T06:54:38.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:38.412+0000 7f4649f12700 1 -- 172.21.15.194:0/2241781306 wait complete. 2024-01-18T06:54:38.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:38.413+0000 7f4649f12700 1 Processor -- start 2024-01-18T06:54:38.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:38.413+0000 7f4649f12700 1 -- start start 2024-01-18T06:54:38.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:38.413+0000 7f4649f12700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46440f5410 0x7f4644108920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:54:38.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:38.413+0000 7f4649f12700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4644108e10 con 0x7f46440f5410 2024-01-18T06:54:38.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:38.413+0000 7f46437fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46440f5410 0x7f4644108920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:54:38.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:38.413+0000 7f46437fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46440f5410 0x7f4644108920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:49852/0 (socket says 172.21.15.194:49852) 2024-01-18T06:54:38.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:38.413+0000 7f46437fe700 1 -- 172.21.15.194:0/3048192335 learned_addr learned my addr 172.21.15.194:0/3048192335 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:54:38.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:38.414+0000 7f46437fe700 1 -- 172.21.15.194:0/3048192335 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4638005040 con 0x7f46440f5410 2024-01-18T06:54:38.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:38.414+0000 7f46437fe700 1 --2- 172.21.15.194:0/3048192335 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46440f5410 0x7f4644108920 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f4638013510 tx=0x7f463800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:54:38.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:38.414+0000 7f462bfff700 1 -- 172.21.15.194:0/3048192335 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4638009320 con 0x7f46440f5410 2024-01-18T06:54:38.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:38.414+0000 7f4649f12700 1 -- 172.21.15.194:0/3048192335 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4644109010 con 0x7f46440f5410 2024-01-18T06:54:38.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:38.414+0000 7f462bfff700 1 -- 172.21.15.194:0/3048192335 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4638013f00 con 0x7f46440f5410 2024-01-18T06:54:38.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:38.414+0000 7f4649f12700 1 -- 172.21.15.194:0/3048192335 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46441094a0 con 0x7f46440f5410 2024-01-18T06:54:38.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:38.414+0000 7f462bfff700 1 -- 172.21.15.194:0/3048192335 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4638014d90 con 0x7f46440f5410 2024-01-18T06:54:38.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:38.415+0000 7f462bfff700 1 -- 172.21.15.194:0/3048192335 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4638014420 con 0x7f46440f5410 2024-01-18T06:54:38.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:38.415+0000 7f4649f12700 1 -- 172.21.15.194:0/3048192335 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4644045ad0 con 0x7f46440f5410 2024-01-18T06:54:38.418 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:38.415+0000 7f462bfff700 1 --2- 172.21.15.194:0/3048192335 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f462c02ffb0 0x7f462c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:54:38.418 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:38.415+0000 7f462bfff700 1 -- 172.21.15.194:0/3048192335 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4638040fd0 con 0x7f46440f5410 2024-01-18T06:54:38.418 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:38.415+0000 7f4642ffd700 1 --2- 172.21.15.194:0/3048192335 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f462c02ffb0 0x7f462c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:54:38.418 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:38.416+0000 7f4642ffd700 1 --2- 172.21.15.194:0/3048192335 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f462c02ffb0 0x7f462c032470 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f46340096a0 tx=0x7f4634006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:54:38.420 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:38.420+0000 7f462bfff700 1 -- 172.21.15.194:0/3048192335 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f463806d2b0 con 0x7f46440f5410 2024-01-18T06:54:38.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:38.628+0000 7f4649f12700 1 -- 172.21.15.194:0/3048192335 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f464410a350 con 0x7f46440f5410 2024-01-18T06:54:38.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:38.629+0000 7f462bfff700 1 -- 172.21.15.194:0/3048192335 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f46380095f0 con 0x7f46440f5410 2024-01-18T06:54:38.629 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:54:38.629 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:54:38.631 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:38.630+0000 7f4649f12700 1 -- 172.21.15.194:0/3048192335 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f462c02ffb0 msgr2=0x7f462c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:38.631 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:38.630+0000 7f4649f12700 1 --2- 172.21.15.194:0/3048192335 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f462c02ffb0 0x7f462c032470 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f46340096a0 tx=0x7f4634006b40).stop 2024-01-18T06:54:38.631 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:38.630+0000 7f4649f12700 1 -- 172.21.15.194:0/3048192335 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46440f5410 msgr2=0x7f4644108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:38.631 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:38.630+0000 7f4649f12700 1 --2- 172.21.15.194:0/3048192335 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46440f5410 0x7f4644108920 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f4638013510 tx=0x7f463800b040).stop 2024-01-18T06:54:38.631 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:38.630+0000 7f4649f12700 1 -- 172.21.15.194:0/3048192335 shutdown_connections 2024-01-18T06:54:38.631 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:38.630+0000 7f4649f12700 1 --2- 172.21.15.194:0/3048192335 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f462c02ffb0 0x7f462c032470 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:38.631 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:38.630+0000 7f4649f12700 1 --2- 172.21.15.194:0/3048192335 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f46440f5410 0x7f4644108920 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:38.631 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:38.630+0000 7f4649f12700 1 -- 172.21.15.194:0/3048192335 >> 172.21.15.194:0/3048192335 conn(0x7f46440f0b70 msgr2=0x7f464405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:54:38.631 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:38.630+0000 7f4649f12700 1 -- 172.21.15.194:0/3048192335 shutdown_connections 2024-01-18T06:54:38.631 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:38.630+0000 7f4649f12700 1 -- 172.21.15.194:0/3048192335 wait complete. 2024-01-18T06:54:38.632 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:54:39.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:38 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:54:38.031868+0000 mgr.smithi047.fuhzzr (mgr.14162) 190 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:54:39.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:38 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:54:38.630659+0000 mon.smithi047 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.194:0/3048192335' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:54:40.015 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:54:40.016 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:54:41.147 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:41.145+0000 7f1584ca2700 1 -- 172.21.15.194:0/2945872293 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1580060280 msgr2=0x7f15800f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:41.147 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:41.145+0000 7f1584ca2700 1 --2- 172.21.15.194:0/2945872293 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1580060280 0x7f15800f90f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f1574009fb0 tx=0x7f1574009b30).stop 2024-01-18T06:54:41.148 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:41.145+0000 7f1584ca2700 1 -- 172.21.15.194:0/2945872293 shutdown_connections 2024-01-18T06:54:41.148 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:41.145+0000 7f1584ca2700 1 --2- 172.21.15.194:0/2945872293 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1580060280 0x7f15800f90f0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:41.148 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:41.145+0000 7f1584ca2700 1 -- 172.21.15.194:0/2945872293 >> 172.21.15.194:0/2945872293 conn(0x7f15800f0b70 msgr2=0x7f15800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:54:41.148 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:41.145+0000 7f1584ca2700 1 -- 172.21.15.194:0/2945872293 shutdown_connections 2024-01-18T06:54:41.148 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:41.145+0000 7f1584ca2700 1 -- 172.21.15.194:0/2945872293 wait complete. 2024-01-18T06:54:41.148 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:41.146+0000 7f1584ca2700 1 Processor -- start 2024-01-18T06:54:41.148 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:41.146+0000 7f1584ca2700 1 -- start start 2024-01-18T06:54:41.148 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:41.146+0000 7f1584ca2700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1580060280 0x7f15801066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:54:41.148 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:41.146+0000 7f1584ca2700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1580106bd0 con 0x7f1580060280 2024-01-18T06:54:41.148 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:41.146+0000 7f157e59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1580060280 0x7f15801066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:54:41.148 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:41.146+0000 7f157e59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1580060280 0x7f15801066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:49872/0 (socket says 172.21.15.194:49872) 2024-01-18T06:54:41.148 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:41.146+0000 7f157e59c700 1 -- 172.21.15.194:0/2260964736 learned_addr learned my addr 172.21.15.194:0/2260964736 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:54:41.148 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:41.147+0000 7f157e59c700 1 -- 172.21.15.194:0/2260964736 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1574005040 con 0x7f1580060280 2024-01-18T06:54:41.148 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:41.147+0000 7f157e59c700 1 --2- 172.21.15.194:0/2260964736 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1580060280 0x7f15801066e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f1574013510 tx=0x7f157400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:54:41.149 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:41.147+0000 7f156effd700 1 -- 172.21.15.194:0/2260964736 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1574009320 con 0x7f1580060280 2024-01-18T06:54:41.149 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:41.147+0000 7f1584ca2700 1 -- 172.21.15.194:0/2260964736 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1580106dd0 con 0x7f1580060280 2024-01-18T06:54:41.149 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:41.147+0000 7f156effd700 1 -- 172.21.15.194:0/2260964736 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1574013f00 con 0x7f1580060280 2024-01-18T06:54:41.149 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:41.147+0000 7f156effd700 1 -- 172.21.15.194:0/2260964736 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1574014dc0 con 0x7f1580060280 2024-01-18T06:54:41.149 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:41.147+0000 7f1584ca2700 1 -- 172.21.15.194:0/2260964736 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1580107260 con 0x7f1580060280 2024-01-18T06:54:41.150 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:41.148+0000 7f156effd700 1 -- 172.21.15.194:0/2260964736 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f15740095f0 con 0x7f1580060280 2024-01-18T06:54:41.150 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:41.148+0000 7f1584ca2700 1 -- 172.21.15.194:0/2260964736 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1580045ad0 con 0x7f1580060280 2024-01-18T06:54:41.150 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:41.148+0000 7f156effd700 1 --2- 172.21.15.194:0/2260964736 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f156802ffb0 0x7f1568032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:54:41.150 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:41.148+0000 7f156effd700 1 -- 172.21.15.194:0/2260964736 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f15740410b0 con 0x7f1580060280 2024-01-18T06:54:41.150 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:41.148+0000 7f157dd9b700 1 --2- 172.21.15.194:0/2260964736 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f156802ffb0 0x7f1568032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:54:41.150 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:41.149+0000 7f157dd9b700 1 --2- 172.21.15.194:0/2260964736 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f156802ffb0 0x7f1568032470 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f15700096a0 tx=0x7f1570006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:54:41.153 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:41.153+0000 7f156effd700 1 -- 172.21.15.194:0/2260964736 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f157406d2b0 con 0x7f1580060280 2024-01-18T06:54:41.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:41.360+0000 7f1584ca2700 1 -- 172.21.15.194:0/2260964736 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1580108110 con 0x7f1580060280 2024-01-18T06:54:41.361 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:41.360+0000 7f156effd700 1 -- 172.21.15.194:0/2260964736 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1574014420 con 0x7f1580060280 2024-01-18T06:54:41.361 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:54:41.361 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:54:41.362 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:41.362+0000 7f1584ca2700 1 -- 172.21.15.194:0/2260964736 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f156802ffb0 msgr2=0x7f1568032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:41.362 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:41.362+0000 7f1584ca2700 1 --2- 172.21.15.194:0/2260964736 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f156802ffb0 0x7f1568032470 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f15700096a0 tx=0x7f1570006b40).stop 2024-01-18T06:54:41.362 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:41.362+0000 7f1584ca2700 1 -- 172.21.15.194:0/2260964736 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1580060280 msgr2=0x7f15801066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:41.363 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:41.362+0000 7f1584ca2700 1 --2- 172.21.15.194:0/2260964736 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1580060280 0x7f15801066e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f1574013510 tx=0x7f157400b040).stop 2024-01-18T06:54:41.363 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:41.362+0000 7f1584ca2700 1 -- 172.21.15.194:0/2260964736 shutdown_connections 2024-01-18T06:54:41.363 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:41.362+0000 7f1584ca2700 1 --2- 172.21.15.194:0/2260964736 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f156802ffb0 0x7f1568032470 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:41.363 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:41.362+0000 7f1584ca2700 1 --2- 172.21.15.194:0/2260964736 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1580060280 0x7f15801066e0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:41.363 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:41.362+0000 7f1584ca2700 1 -- 172.21.15.194:0/2260964736 >> 172.21.15.194:0/2260964736 conn(0x7f15800f0b70 msgr2=0x7f15800f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:54:41.363 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:41.362+0000 7f1584ca2700 1 -- 172.21.15.194:0/2260964736 shutdown_connections 2024-01-18T06:54:41.363 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:41.362+0000 7f1584ca2700 1 -- 172.21.15.194:0/2260964736 wait complete. 2024-01-18T06:54:41.363 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:54:41.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:41 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:54:40.032266+0000 mgr.smithi047.fuhzzr (mgr.14162) 191 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:54:42.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:42 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:54:41.362320+0000 mon.smithi047 (mon.0) 259 : audit [DBG] from='client.? 172.21.15.194:0/2260964736' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:54:42.753 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:54:42.754 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:54:43.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:43 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:54:42.032655+0000 mgr.smithi047.fuhzzr (mgr.14162) 192 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:54:43.925 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:43.924+0000 7ff0ebfff700 1 -- 172.21.15.194:0/1917157803 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff0ec0f51a0 msgr2=0x7ff0ec0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:43.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:43.924+0000 7ff0ebfff700 1 --2- 172.21.15.194:0/1917157803 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff0ec0f51a0 0x7ff0ec0f55c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7ff0dc009fb0 tx=0x7ff0dc009b30).stop 2024-01-18T06:54:43.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:43.924+0000 7ff0ebfff700 1 -- 172.21.15.194:0/1917157803 shutdown_connections 2024-01-18T06:54:43.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:43.924+0000 7ff0ebfff700 1 --2- 172.21.15.194:0/1917157803 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff0ec0f51a0 0x7ff0ec0f55c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:43.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:43.924+0000 7ff0ebfff700 1 -- 172.21.15.194:0/1917157803 >> 172.21.15.194:0/1917157803 conn(0x7ff0ec0f09a0 msgr2=0x7ff0ec0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:54:43.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:43.924+0000 7ff0ebfff700 1 -- 172.21.15.194:0/1917157803 shutdown_connections 2024-01-18T06:54:43.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:43.924+0000 7ff0ebfff700 1 -- 172.21.15.194:0/1917157803 wait complete. 2024-01-18T06:54:43.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:43.925+0000 7ff0ebfff700 1 Processor -- start 2024-01-18T06:54:43.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:43.925+0000 7ff0ebfff700 1 -- start start 2024-01-18T06:54:43.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:43.925+0000 7ff0ebfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff0ec0f51a0 0x7ff0ec1091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:54:43.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:43.925+0000 7ff0ebfff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0ec060240 con 0x7ff0ec0f51a0 2024-01-18T06:54:43.928 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:43.926+0000 7ff0eaffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff0ec0f51a0 0x7ff0ec1091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:54:43.928 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:43.926+0000 7ff0eaffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff0ec0f51a0 0x7ff0ec1091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:49892/0 (socket says 172.21.15.194:49892) 2024-01-18T06:54:43.928 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:43.926+0000 7ff0eaffd700 1 -- 172.21.15.194:0/4130311996 learned_addr learned my addr 172.21.15.194:0/4130311996 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:54:43.928 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:43.926+0000 7ff0eaffd700 1 -- 172.21.15.194:0/4130311996 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0dc005040 con 0x7ff0ec0f51a0 2024-01-18T06:54:43.928 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:43.926+0000 7ff0eaffd700 1 --2- 172.21.15.194:0/4130311996 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff0ec0f51a0 0x7ff0ec1091b0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7ff0dc013510 tx=0x7ff0dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:54:43.928 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:43.926+0000 7ff0d37fe700 1 -- 172.21.15.194:0/4130311996 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0dc009320 con 0x7ff0ec0f51a0 2024-01-18T06:54:43.928 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:43.926+0000 7ff0ebfff700 1 -- 172.21.15.194:0/4130311996 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0ec060440 con 0x7ff0ec0f51a0 2024-01-18T06:54:43.928 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:43.926+0000 7ff0ebfff700 1 -- 172.21.15.194:0/4130311996 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0ec109860 con 0x7ff0ec0f51a0 2024-01-18T06:54:43.928 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:43.926+0000 7ff0d37fe700 1 -- 172.21.15.194:0/4130311996 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff0dc013f00 con 0x7ff0ec0f51a0 2024-01-18T06:54:43.928 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:43.926+0000 7ff0d37fe700 1 -- 172.21.15.194:0/4130311996 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0dc014d90 con 0x7ff0ec0f51a0 2024-01-18T06:54:43.929 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:43.927+0000 7ff0d37fe700 1 -- 172.21.15.194:0/4130311996 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff0dc0095f0 con 0x7ff0ec0f51a0 2024-01-18T06:54:43.929 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:43.927+0000 7ff0ebfff700 1 -- 172.21.15.194:0/4130311996 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0cc004fa0 con 0x7ff0ec0f51a0 2024-01-18T06:54:43.929 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:43.927+0000 7ff0d37fe700 1 --2- 172.21.15.194:0/4130311996 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7ff0d402ff60 0x7ff0d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:54:43.929 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:43.927+0000 7ff0d37fe700 1 -- 172.21.15.194:0/4130311996 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff0dc040fc0 con 0x7ff0ec0f51a0 2024-01-18T06:54:43.929 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:43.928+0000 7ff0ea7fc700 1 --2- 172.21.15.194:0/4130311996 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7ff0d402ff60 0x7ff0d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:54:43.929 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:43.928+0000 7ff0ea7fc700 1 --2- 172.21.15.194:0/4130311996 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7ff0d402ff60 0x7ff0d4032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7ff0e00096a0 tx=0x7ff0e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:54:43.932 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:43.932+0000 7ff0d37fe700 1 -- 172.21.15.194:0/4130311996 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff0dc06d2b0 con 0x7ff0ec0f51a0 2024-01-18T06:54:44.140 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:44.140+0000 7ff0ebfff700 1 -- 172.21.15.194:0/4130311996 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff0cc005670 con 0x7ff0ec0f51a0 2024-01-18T06:54:44.141 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:44.140+0000 7ff0d37fe700 1 -- 172.21.15.194:0/4130311996 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff0dc014420 con 0x7ff0ec0f51a0 2024-01-18T06:54:44.141 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:54:44.141 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:54:44.142 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:44.142+0000 7ff0ebfff700 1 -- 172.21.15.194:0/4130311996 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7ff0d402ff60 msgr2=0x7ff0d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:44.142 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:44.142+0000 7ff0ebfff700 1 --2- 172.21.15.194:0/4130311996 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7ff0d402ff60 0x7ff0d4032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7ff0e00096a0 tx=0x7ff0e0006b40).stop 2024-01-18T06:54:44.143 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:44.142+0000 7ff0ebfff700 1 -- 172.21.15.194:0/4130311996 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff0ec0f51a0 msgr2=0x7ff0ec1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:44.143 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:44.142+0000 7ff0ebfff700 1 --2- 172.21.15.194:0/4130311996 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff0ec0f51a0 0x7ff0ec1091b0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7ff0dc013510 tx=0x7ff0dc00b040).stop 2024-01-18T06:54:44.143 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:44.142+0000 7ff0ebfff700 1 -- 172.21.15.194:0/4130311996 shutdown_connections 2024-01-18T06:54:44.143 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:44.142+0000 7ff0ebfff700 1 --2- 172.21.15.194:0/4130311996 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7ff0d402ff60 0x7ff0d4032420 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:44.143 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:44.142+0000 7ff0ebfff700 1 --2- 172.21.15.194:0/4130311996 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff0ec0f51a0 0x7ff0ec1091b0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:44.143 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:44.142+0000 7ff0ebfff700 1 -- 172.21.15.194:0/4130311996 >> 172.21.15.194:0/4130311996 conn(0x7ff0ec0f09a0 msgr2=0x7ff0ec0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:54:44.143 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:44.142+0000 7ff0ebfff700 1 -- 172.21.15.194:0/4130311996 shutdown_connections 2024-01-18T06:54:44.143 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:44.142+0000 7ff0ebfff700 1 -- 172.21.15.194:0/4130311996 wait complete. 2024-01-18T06:54:44.143 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:54:45.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:45 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:54:44.033033+0000 mgr.smithi047.fuhzzr (mgr.14162) 193 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:54:45.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:45 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:54:44.142417+0000 mon.smithi047 (mon.0) 260 : audit [DBG] from='client.? 172.21.15.194:0/4130311996' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:54:45.539 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:54:45.540 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:54:46.813 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:46.813+0000 7f6dfc28a700 1 -- 172.21.15.194:0/1688919655 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6df40f5410 msgr2=0x7f6df40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:46.813 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:46.813+0000 7f6dfc28a700 1 --2- 172.21.15.194:0/1688919655 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6df40f5410 0x7f6df40f5830 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f6de8009fb0 tx=0x7f6de8009b30).stop 2024-01-18T06:54:46.813 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:46.813+0000 7f6dfc28a700 1 -- 172.21.15.194:0/1688919655 shutdown_connections 2024-01-18T06:54:46.813 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:46.813+0000 7f6dfc28a700 1 --2- 172.21.15.194:0/1688919655 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6df40f5410 0x7f6df40f5830 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:46.813 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:46.813+0000 7f6dfc28a700 1 -- 172.21.15.194:0/1688919655 >> 172.21.15.194:0/1688919655 conn(0x7f6df40f0b70 msgr2=0x7f6df40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:54:46.813 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:46.813+0000 7f6dfc28a700 1 -- 172.21.15.194:0/1688919655 shutdown_connections 2024-01-18T06:54:46.813 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:46.813+0000 7f6dfc28a700 1 -- 172.21.15.194:0/1688919655 wait complete. 2024-01-18T06:54:46.813 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:46.813+0000 7f6dfc28a700 1 Processor -- start 2024-01-18T06:54:46.813 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:46.813+0000 7f6dfc28a700 1 -- start start 2024-01-18T06:54:46.813 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:46.813+0000 7f6dfc28a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6df40f5410 0x7f6df4106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:54:46.813 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:46.813+0000 7f6dfc28a700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df4106c10 con 0x7f6df40f5410 2024-01-18T06:54:46.815 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:46.814+0000 7f6dfa026700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6df40f5410 0x7f6df4106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:54:46.815 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:46.814+0000 7f6dfa026700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6df40f5410 0x7f6df4106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:49920/0 (socket says 172.21.15.194:49920) 2024-01-18T06:54:46.815 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:46.814+0000 7f6dfa026700 1 -- 172.21.15.194:0/1741573584 learned_addr learned my addr 172.21.15.194:0/1741573584 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:54:46.815 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:46.814+0000 7f6dfa026700 1 -- 172.21.15.194:0/1741573584 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de8005040 con 0x7f6df40f5410 2024-01-18T06:54:46.815 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:46.814+0000 7f6dfa026700 1 --2- 172.21.15.194:0/1741573584 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6df40f5410 0x7f6df4106720 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f6de8014000 tx=0x7f6de800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:54:46.815 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:46.814+0000 7f6de6ffd700 1 -- 172.21.15.194:0/1741573584 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6de80095f0 con 0x7f6df40f5410 2024-01-18T06:54:46.815 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:46.814+0000 7f6dfc28a700 1 -- 172.21.15.194:0/1741573584 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6df4106e10 con 0x7f6df40f5410 2024-01-18T06:54:46.815 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:46.814+0000 7f6de6ffd700 1 -- 172.21.15.194:0/1741573584 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6de8007e70 con 0x7f6df40f5410 2024-01-18T06:54:46.816 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:46.814+0000 7f6de6ffd700 1 -- 172.21.15.194:0/1741573584 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6de8014ea0 con 0x7f6df40f5410 2024-01-18T06:54:46.816 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:46.814+0000 7f6dfc28a700 1 -- 172.21.15.194:0/1741573584 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6df41072a0 con 0x7f6df40f5410 2024-01-18T06:54:46.816 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:46.815+0000 7f6de6ffd700 1 -- 172.21.15.194:0/1741573584 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6de8025070 con 0x7f6df40f5410 2024-01-18T06:54:46.816 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:46.815+0000 7f6dfc28a700 1 -- 172.21.15.194:0/1741573584 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6df4045ad0 con 0x7f6df40f5410 2024-01-18T06:54:46.816 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:46.815+0000 7f6de6ffd700 1 --2- 172.21.15.194:0/1741573584 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f6de002ff60 0x7f6de0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:54:46.816 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:46.815+0000 7f6de6ffd700 1 -- 172.21.15.194:0/1741573584 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6de8040930 con 0x7f6df40f5410 2024-01-18T06:54:46.816 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:46.816+0000 7f6df9825700 1 --2- 172.21.15.194:0/1741573584 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f6de002ff60 0x7f6de0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:54:46.816 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:46.816+0000 7f6df9825700 1 --2- 172.21.15.194:0/1741573584 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f6de002ff60 0x7f6de0032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f6df00096a0 tx=0x7f6df0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:54:46.820 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:46.820+0000 7f6de6ffd700 1 -- 172.21.15.194:0/1741573584 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6de800f4e0 con 0x7f6df40f5410 2024-01-18T06:54:47.024 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:47.024+0000 7f6dfc28a700 1 -- 172.21.15.194:0/1741573584 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6df4108150 con 0x7f6df40f5410 2024-01-18T06:54:47.025 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:47.024+0000 7f6de6ffd700 1 -- 172.21.15.194:0/1741573584 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6de800f300 con 0x7f6df40f5410 2024-01-18T06:54:47.026 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:54:47.026 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:54:47.026 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:47.026+0000 7f6dfc28a700 1 -- 172.21.15.194:0/1741573584 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f6de002ff60 msgr2=0x7f6de0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:47.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:47.026+0000 7f6dfc28a700 1 --2- 172.21.15.194:0/1741573584 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f6de002ff60 0x7f6de0032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f6df00096a0 tx=0x7f6df0006b40).stop 2024-01-18T06:54:47.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:47.026+0000 7f6dfc28a700 1 -- 172.21.15.194:0/1741573584 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6df40f5410 msgr2=0x7f6df4106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:47.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:47.026+0000 7f6dfc28a700 1 --2- 172.21.15.194:0/1741573584 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6df40f5410 0x7f6df4106720 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f6de8014000 tx=0x7f6de800b040).stop 2024-01-18T06:54:47.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:47.026+0000 7f6dfc28a700 1 -- 172.21.15.194:0/1741573584 shutdown_connections 2024-01-18T06:54:47.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:47.026+0000 7f6dfc28a700 1 --2- 172.21.15.194:0/1741573584 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f6de002ff60 0x7f6de0032420 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:47.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:47.026+0000 7f6dfc28a700 1 --2- 172.21.15.194:0/1741573584 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6df40f5410 0x7f6df4106720 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:47.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:47.026+0000 7f6dfc28a700 1 -- 172.21.15.194:0/1741573584 >> 172.21.15.194:0/1741573584 conn(0x7f6df40f0b70 msgr2=0x7f6df405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:54:47.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:47.026+0000 7f6dfc28a700 1 -- 172.21.15.194:0/1741573584 shutdown_connections 2024-01-18T06:54:47.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:47.026+0000 7f6dfc28a700 1 -- 172.21.15.194:0/1741573584 wait complete. 2024-01-18T06:54:47.027 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:54:47.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:47 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:54:46.033431+0000 mgr.smithi047.fuhzzr (mgr.14162) 194 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:54:47.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:47 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:54:47.025474+0000 mon.smithi047 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.194:0/1741573584' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:54:48.425 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:54:48.425 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:54:49.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:48 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:54:48.033761+0000 mgr.smithi047.fuhzzr (mgr.14162) 195 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:54:49.577 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:49.577+0000 7fb818906700 1 -- 172.21.15.194:0/400717160 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8140f6510 msgr2=0x7fb8140f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:49.577 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:49.577+0000 7fb818906700 1 --2- 172.21.15.194:0/400717160 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8140f6510 0x7fb8140f6930 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7fb808009fb0 tx=0x7fb808009b30).stop 2024-01-18T06:54:49.577 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:49.577+0000 7fb818906700 1 -- 172.21.15.194:0/400717160 shutdown_connections 2024-01-18T06:54:49.577 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:49.577+0000 7fb818906700 1 --2- 172.21.15.194:0/400717160 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8140f6510 0x7fb8140f6930 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:49.577 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:49.577+0000 7fb818906700 1 -- 172.21.15.194:0/400717160 >> 172.21.15.194:0/400717160 conn(0x7fb8140f1c70 msgr2=0x7fb8140f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:54:49.578 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:49.577+0000 7fb818906700 1 -- 172.21.15.194:0/400717160 shutdown_connections 2024-01-18T06:54:49.578 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:49.577+0000 7fb818906700 1 -- 172.21.15.194:0/400717160 wait complete. 2024-01-18T06:54:49.580 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:49.578+0000 7fb818906700 1 Processor -- start 2024-01-18T06:54:49.580 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:49.578+0000 7fb818906700 1 -- start start 2024-01-18T06:54:49.580 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:49.578+0000 7fb818906700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8140f6510 0x7fb814106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:54:49.580 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:49.578+0000 7fb818906700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb814106c20 con 0x7fb8140f6510 2024-01-18T06:54:49.580 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:49.579+0000 7fb81259c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8140f6510 0x7fb814106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:54:49.580 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:49.579+0000 7fb81259c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8140f6510 0x7fb814106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:55366/0 (socket says 172.21.15.194:55366) 2024-01-18T06:54:49.580 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:49.579+0000 7fb81259c700 1 -- 172.21.15.194:0/2124243210 learned_addr learned my addr 172.21.15.194:0/2124243210 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:54:49.580 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:49.579+0000 7fb81259c700 1 -- 172.21.15.194:0/2124243210 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb808005040 con 0x7fb8140f6510 2024-01-18T06:54:49.580 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:49.579+0000 7fb81259c700 1 --2- 172.21.15.194:0/2124243210 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8140f6510 0x7fb814106730 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fb808013510 tx=0x7fb80800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:54:49.580 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:49.579+0000 7fb802ffd700 1 -- 172.21.15.194:0/2124243210 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb808009320 con 0x7fb8140f6510 2024-01-18T06:54:49.581 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:49.579+0000 7fb818906700 1 -- 172.21.15.194:0/2124243210 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb814106e20 con 0x7fb8140f6510 2024-01-18T06:54:49.581 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:49.579+0000 7fb802ffd700 1 -- 172.21.15.194:0/2124243210 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb808013f00 con 0x7fb8140f6510 2024-01-18T06:54:49.581 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:49.579+0000 7fb802ffd700 1 -- 172.21.15.194:0/2124243210 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb808014d90 con 0x7fb8140f6510 2024-01-18T06:54:49.581 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:49.579+0000 7fb818906700 1 -- 172.21.15.194:0/2124243210 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8141072b0 con 0x7fb8140f6510 2024-01-18T06:54:49.581 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:49.580+0000 7fb802ffd700 1 -- 172.21.15.194:0/2124243210 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb8080095f0 con 0x7fb8140f6510 2024-01-18T06:54:49.581 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:49.580+0000 7fb818906700 1 -- 172.21.15.194:0/2124243210 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb814045ad0 con 0x7fb8140f6510 2024-01-18T06:54:49.581 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:49.580+0000 7fb802ffd700 1 --2- 172.21.15.194:0/2124243210 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb7fc02ff60 0x7fb7fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:54:49.581 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:49.580+0000 7fb802ffd700 1 -- 172.21.15.194:0/2124243210 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb8080410a0 con 0x7fb8140f6510 2024-01-18T06:54:49.581 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:49.580+0000 7fb811d9b700 1 --2- 172.21.15.194:0/2124243210 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb7fc02ff60 0x7fb7fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:54:49.581 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:49.581+0000 7fb811d9b700 1 --2- 172.21.15.194:0/2124243210 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb7fc02ff60 0x7fb7fc032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fb8040096a0 tx=0x7fb804006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:54:49.584 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:49.585+0000 7fb802ffd700 1 -- 172.21.15.194:0/2124243210 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb80801d7f0 con 0x7fb8140f6510 2024-01-18T06:54:49.791 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:49.791+0000 7fb818906700 1 -- 172.21.15.194:0/2124243210 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb8140fbf60 con 0x7fb8140f6510 2024-01-18T06:54:49.792 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:49.792+0000 7fb802ffd700 1 -- 172.21.15.194:0/2124243210 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb808010230 con 0x7fb8140f6510 2024-01-18T06:54:49.792 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:54:49.792 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:54:49.793 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:49.794+0000 7fb818906700 1 -- 172.21.15.194:0/2124243210 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb7fc02ff60 msgr2=0x7fb7fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:49.793 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:49.794+0000 7fb818906700 1 --2- 172.21.15.194:0/2124243210 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb7fc02ff60 0x7fb7fc032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fb8040096a0 tx=0x7fb804006b40).stop 2024-01-18T06:54:49.793 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:49.794+0000 7fb818906700 1 -- 172.21.15.194:0/2124243210 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8140f6510 msgr2=0x7fb814106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:49.793 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:49.794+0000 7fb818906700 1 --2- 172.21.15.194:0/2124243210 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8140f6510 0x7fb814106730 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fb808013510 tx=0x7fb80800b040).stop 2024-01-18T06:54:49.794 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:49.794+0000 7fb818906700 1 -- 172.21.15.194:0/2124243210 shutdown_connections 2024-01-18T06:54:49.794 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:49.794+0000 7fb818906700 1 --2- 172.21.15.194:0/2124243210 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb7fc02ff60 0x7fb7fc032420 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:49.794 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:49.794+0000 7fb818906700 1 --2- 172.21.15.194:0/2124243210 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8140f6510 0x7fb814106730 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:49.794 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:49.794+0000 7fb818906700 1 -- 172.21.15.194:0/2124243210 >> 172.21.15.194:0/2124243210 conn(0x7fb8140f1c70 msgr2=0x7fb8140f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:54:49.794 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:49.794+0000 7fb818906700 1 -- 172.21.15.194:0/2124243210 shutdown_connections 2024-01-18T06:54:49.794 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:49.794+0000 7fb818906700 1 -- 172.21.15.194:0/2124243210 wait complete. 2024-01-18T06:54:49.794 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:54:50.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:49 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:54:49.793192+0000 mon.smithi047 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.194:0/2124243210' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:54:51.172 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:54:51.173 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:54:51.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:50 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:54:50.034191+0000 mgr.smithi047.fuhzzr (mgr.14162) 196 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:54:52.367 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:52.367+0000 7efde79d9700 1 -- 172.21.15.194:0/2932094251 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efde00f5410 msgr2=0x7efde00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:52.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:52.367+0000 7efde79d9700 1 --2- 172.21.15.194:0/2932094251 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efde00f5410 0x7efde00f5830 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7efdd0009fb0 tx=0x7efdd0009b30).stop 2024-01-18T06:54:52.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:52.367+0000 7efde79d9700 1 -- 172.21.15.194:0/2932094251 shutdown_connections 2024-01-18T06:54:52.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:52.367+0000 7efde79d9700 1 --2- 172.21.15.194:0/2932094251 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efde00f5410 0x7efde00f5830 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:52.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:52.367+0000 7efde79d9700 1 -- 172.21.15.194:0/2932094251 >> 172.21.15.194:0/2932094251 conn(0x7efde00f0b70 msgr2=0x7efde00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:54:52.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:52.367+0000 7efde79d9700 1 -- 172.21.15.194:0/2932094251 shutdown_connections 2024-01-18T06:54:52.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:52.367+0000 7efde79d9700 1 -- 172.21.15.194:0/2932094251 wait complete. 2024-01-18T06:54:52.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:52.368+0000 7efde79d9700 1 Processor -- start 2024-01-18T06:54:52.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:52.368+0000 7efde79d9700 1 -- start start 2024-01-18T06:54:52.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:52.368+0000 7efde79d9700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efde00f5410 0x7efde0106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:54:52.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:52.368+0000 7efde79d9700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde0106c10 con 0x7efde00f5410 2024-01-18T06:54:52.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:52.368+0000 7efde5775700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efde00f5410 0x7efde0106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:54:52.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:52.368+0000 7efde5775700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efde00f5410 0x7efde0106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:55386/0 (socket says 172.21.15.194:55386) 2024-01-18T06:54:52.369 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:52.368+0000 7efde5775700 1 -- 172.21.15.194:0/3553194494 learned_addr learned my addr 172.21.15.194:0/3553194494 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:54:52.369 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:52.368+0000 7efde5775700 1 -- 172.21.15.194:0/3553194494 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd0005040 con 0x7efde00f5410 2024-01-18T06:54:52.370 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:52.368+0000 7efde5775700 1 --2- 172.21.15.194:0/3553194494 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efde00f5410 0x7efde0106720 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7efdd0013510 tx=0x7efdd000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:54:52.370 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:52.369+0000 7efdd5ffb700 1 -- 172.21.15.194:0/3553194494 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdd0009320 con 0x7efde00f5410 2024-01-18T06:54:52.370 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:52.369+0000 7efde79d9700 1 -- 172.21.15.194:0/3553194494 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efde0106e10 con 0x7efde00f5410 2024-01-18T06:54:52.370 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:52.369+0000 7efdd5ffb700 1 -- 172.21.15.194:0/3553194494 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efdd0013f00 con 0x7efde00f5410 2024-01-18T06:54:52.370 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:52.369+0000 7efdd5ffb700 1 -- 172.21.15.194:0/3553194494 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdd0014d90 con 0x7efde00f5410 2024-01-18T06:54:52.370 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:52.369+0000 7efde79d9700 1 -- 172.21.15.194:0/3553194494 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efde01072a0 con 0x7efde00f5410 2024-01-18T06:54:52.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:52.370+0000 7efdd5ffb700 1 -- 172.21.15.194:0/3553194494 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efdd00095f0 con 0x7efde00f5410 2024-01-18T06:54:52.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:52.370+0000 7efde79d9700 1 -- 172.21.15.194:0/3553194494 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efde0045ad0 con 0x7efde00f5410 2024-01-18T06:54:52.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:52.370+0000 7efdd5ffb700 1 --2- 172.21.15.194:0/3553194494 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7efdcc02ff60 0x7efdcc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:54:52.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:52.370+0000 7efdd5ffb700 1 -- 172.21.15.194:0/3553194494 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efdd0041080 con 0x7efde00f5410 2024-01-18T06:54:52.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:52.370+0000 7efde4f74700 1 --2- 172.21.15.194:0/3553194494 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7efdcc02ff60 0x7efdcc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:54:52.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:52.370+0000 7efde4f74700 1 --2- 172.21.15.194:0/3553194494 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7efdcc02ff60 0x7efdcc032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7efddc0096a0 tx=0x7efddc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:54:52.373 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:52.374+0000 7efdd5ffb700 1 -- 172.21.15.194:0/3553194494 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efdd0014420 con 0x7efde00f5410 2024-01-18T06:54:52.581 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:52.581+0000 7efde79d9700 1 -- 172.21.15.194:0/3553194494 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efde0108150 con 0x7efde00f5410 2024-01-18T06:54:52.582 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:52.582+0000 7efdd5ffb700 1 -- 172.21.15.194:0/3553194494 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efdd0010230 con 0x7efde00f5410 2024-01-18T06:54:52.582 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:54:52.582 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:54:52.583 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:52.584+0000 7efde79d9700 1 -- 172.21.15.194:0/3553194494 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7efdcc02ff60 msgr2=0x7efdcc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:52.583 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:52.584+0000 7efde79d9700 1 --2- 172.21.15.194:0/3553194494 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7efdcc02ff60 0x7efdcc032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7efddc0096a0 tx=0x7efddc006b40).stop 2024-01-18T06:54:52.583 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:52.584+0000 7efde79d9700 1 -- 172.21.15.194:0/3553194494 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efde00f5410 msgr2=0x7efde0106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:52.584 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:52.584+0000 7efde79d9700 1 --2- 172.21.15.194:0/3553194494 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efde00f5410 0x7efde0106720 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7efdd0013510 tx=0x7efdd000b040).stop 2024-01-18T06:54:52.584 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:52.584+0000 7efde79d9700 1 -- 172.21.15.194:0/3553194494 shutdown_connections 2024-01-18T06:54:52.584 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:52.584+0000 7efde79d9700 1 --2- 172.21.15.194:0/3553194494 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7efdcc02ff60 0x7efdcc032420 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:52.584 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:52.584+0000 7efde79d9700 1 --2- 172.21.15.194:0/3553194494 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efde00f5410 0x7efde0106720 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:52.584 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:52.584+0000 7efde79d9700 1 -- 172.21.15.194:0/3553194494 >> 172.21.15.194:0/3553194494 conn(0x7efde00f0b70 msgr2=0x7efde005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:54:52.584 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:52.584+0000 7efde79d9700 1 -- 172.21.15.194:0/3553194494 shutdown_connections 2024-01-18T06:54:52.584 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:52.584+0000 7efde79d9700 1 -- 172.21.15.194:0/3553194494 wait complete. 2024-01-18T06:54:52.584 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:54:53.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:53 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:54:52.034608+0000 mgr.smithi047.fuhzzr (mgr.14162) 197 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:54:53.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:53 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:54:52.583106+0000 mon.smithi047 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.194:0/3553194494' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:54:53.964 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:54:53.964 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:54:55.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:55.128+0000 7f6f50c2b700 1 -- 172.21.15.194:0/1183105852 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6f4c0f5410 msgr2=0x7f6f4c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:55.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:55.128+0000 7f6f50c2b700 1 --2- 172.21.15.194:0/1183105852 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6f4c0f5410 0x7f6f4c0f5830 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f6f34009fb0 tx=0x7f6f34009b30).stop 2024-01-18T06:54:55.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:55.128+0000 7f6f50c2b700 1 -- 172.21.15.194:0/1183105852 shutdown_connections 2024-01-18T06:54:55.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:55.128+0000 7f6f50c2b700 1 --2- 172.21.15.194:0/1183105852 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6f4c0f5410 0x7f6f4c0f5830 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:55.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:55.128+0000 7f6f50c2b700 1 -- 172.21.15.194:0/1183105852 >> 172.21.15.194:0/1183105852 conn(0x7f6f4c0f0b70 msgr2=0x7f6f4c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:54:55.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:55.128+0000 7f6f50c2b700 1 -- 172.21.15.194:0/1183105852 shutdown_connections 2024-01-18T06:54:55.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:55.128+0000 7f6f50c2b700 1 -- 172.21.15.194:0/1183105852 wait complete. 2024-01-18T06:54:55.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:55.129+0000 7f6f50c2b700 1 Processor -- start 2024-01-18T06:54:55.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:55.129+0000 7f6f50c2b700 1 -- start start 2024-01-18T06:54:55.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:55.129+0000 7f6f50c2b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6f4c0f5410 0x7f6f4c0fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:54:55.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:55.129+0000 7f6f50c2b700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4c0fb820 con 0x7f6f4c0f5410 2024-01-18T06:54:55.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:55.129+0000 7f6f4a59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6f4c0f5410 0x7f6f4c0fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:54:55.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:55.129+0000 7f6f4a59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6f4c0f5410 0x7f6f4c0fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:55408/0 (socket says 172.21.15.194:55408) 2024-01-18T06:54:55.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:55.129+0000 7f6f4a59c700 1 -- 172.21.15.194:0/1579869310 learned_addr learned my addr 172.21.15.194:0/1579869310 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:54:55.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:55.129+0000 7f6f4a59c700 1 -- 172.21.15.194:0/1579869310 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f34005040 con 0x7f6f4c0f5410 2024-01-18T06:54:55.131 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:55.129+0000 7f6f4a59c700 1 --2- 172.21.15.194:0/1579869310 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6f4c0f5410 0x7f6f4c0fd3a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f6f34013510 tx=0x7f6f3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:54:55.131 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:55.130+0000 7f6f42ffd700 1 -- 172.21.15.194:0/1579869310 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f34009320 con 0x7f6f4c0f5410 2024-01-18T06:54:55.131 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:55.130+0000 7f6f50c2b700 1 -- 172.21.15.194:0/1579869310 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f4c0fb9c0 con 0x7f6f4c0f5410 2024-01-18T06:54:55.131 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:55.130+0000 7f6f42ffd700 1 -- 172.21.15.194:0/1579869310 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6f34013f00 con 0x7f6f4c0f5410 2024-01-18T06:54:55.131 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:55.130+0000 7f6f42ffd700 1 -- 172.21.15.194:0/1579869310 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f34014d90 con 0x7f6f4c0f5410 2024-01-18T06:54:55.131 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:55.130+0000 7f6f50c2b700 1 -- 172.21.15.194:0/1579869310 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f4c0fbe50 con 0x7f6f4c0f5410 2024-01-18T06:54:55.131 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:55.130+0000 7f6f42ffd700 1 -- 172.21.15.194:0/1579869310 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6f340263a0 con 0x7f6f4c0f5410 2024-01-18T06:54:55.131 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:55.131+0000 7f6f42ffd700 1 --2- 172.21.15.194:0/1579869310 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f6f3802ff60 0x7f6f38032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:54:55.131 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:55.131+0000 7f6f50c2b700 1 -- 172.21.15.194:0/1579869310 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f4c059100 con 0x7f6f4c0f5410 2024-01-18T06:54:55.131 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:55.131+0000 7f6f42ffd700 1 -- 172.21.15.194:0/1579869310 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6f340410c0 con 0x7f6f4c0f5410 2024-01-18T06:54:55.131 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:55.131+0000 7f6f49d9b700 1 --2- 172.21.15.194:0/1579869310 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f6f3802ff60 0x7f6f38032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:54:55.132 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:55.131+0000 7f6f49d9b700 1 --2- 172.21.15.194:0/1579869310 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f6f3802ff60 0x7f6f38032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f6f3c0096a0 tx=0x7f6f3c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:54:55.135 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:55.135+0000 7f6f42ffd700 1 -- 172.21.15.194:0/1579869310 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6f34012920 con 0x7f6f4c0f5410 2024-01-18T06:54:55.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:55.342+0000 7f6f50c2b700 1 -- 172.21.15.194:0/1579869310 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6f4c0fae60 con 0x7f6f4c0f5410 2024-01-18T06:54:55.342 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:55.343+0000 7f6f42ffd700 1 -- 172.21.15.194:0/1579869310 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6f3402b020 con 0x7f6f4c0f5410 2024-01-18T06:54:55.342 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:54:55.342 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:54:55.344 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:55.344+0000 7f6f50c2b700 1 -- 172.21.15.194:0/1579869310 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f6f3802ff60 msgr2=0x7f6f38032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:55.344 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:55.344+0000 7f6f50c2b700 1 --2- 172.21.15.194:0/1579869310 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f6f3802ff60 0x7f6f38032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f6f3c0096a0 tx=0x7f6f3c006b40).stop 2024-01-18T06:54:55.344 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:55.344+0000 7f6f50c2b700 1 -- 172.21.15.194:0/1579869310 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6f4c0f5410 msgr2=0x7f6f4c0fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:55.344 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:55.344+0000 7f6f50c2b700 1 --2- 172.21.15.194:0/1579869310 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6f4c0f5410 0x7f6f4c0fd3a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f6f34013510 tx=0x7f6f3400b040).stop 2024-01-18T06:54:55.344 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:55.344+0000 7f6f50c2b700 1 -- 172.21.15.194:0/1579869310 shutdown_connections 2024-01-18T06:54:55.344 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:55.344+0000 7f6f50c2b700 1 --2- 172.21.15.194:0/1579869310 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f6f3802ff60 0x7f6f38032420 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:55.344 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:55.344+0000 7f6f50c2b700 1 --2- 172.21.15.194:0/1579869310 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6f4c0f5410 0x7f6f4c0fd3a0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:55.344 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:55.344+0000 7f6f50c2b700 1 -- 172.21.15.194:0/1579869310 >> 172.21.15.194:0/1579869310 conn(0x7f6f4c0f0b70 msgr2=0x7f6f4c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:54:55.344 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:55.344+0000 7f6f50c2b700 1 -- 172.21.15.194:0/1579869310 shutdown_connections 2024-01-18T06:54:55.345 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:55.344+0000 7f6f50c2b700 1 -- 172.21.15.194:0/1579869310 wait complete. 2024-01-18T06:54:55.345 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:54:55.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:55 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:54:54.034945+0000 mgr.smithi047.fuhzzr (mgr.14162) 198 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:54:56.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:54:55.343707+0000 mon.smithi047 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.194:0/1579869310' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:54:56.737 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:54:56.738 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:54:57.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:57 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:54:56.035254+0000 mgr.smithi047.fuhzzr (mgr.14162) 199 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:54:57.950 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:57.950+0000 7faf91880700 1 -- 172.21.15.194:0/3311346181 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf8c0f5410 msgr2=0x7faf8c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:57.950 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:57.950+0000 7faf91880700 1 --2- 172.21.15.194:0/3311346181 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf8c0f5410 0x7faf8c0f5830 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7faf80009fb0 tx=0x7faf80009b30).stop 2024-01-18T06:54:57.950 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:57.950+0000 7faf91880700 1 -- 172.21.15.194:0/3311346181 shutdown_connections 2024-01-18T06:54:57.950 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:57.950+0000 7faf91880700 1 --2- 172.21.15.194:0/3311346181 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf8c0f5410 0x7faf8c0f5830 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:57.950 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:57.950+0000 7faf91880700 1 -- 172.21.15.194:0/3311346181 >> 172.21.15.194:0/3311346181 conn(0x7faf8c0f0b70 msgr2=0x7faf8c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:54:57.950 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:57.950+0000 7faf91880700 1 -- 172.21.15.194:0/3311346181 shutdown_connections 2024-01-18T06:54:57.950 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:57.950+0000 7faf91880700 1 -- 172.21.15.194:0/3311346181 wait complete. 2024-01-18T06:54:57.951 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:57.951+0000 7faf91880700 1 Processor -- start 2024-01-18T06:54:57.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:57.951+0000 7faf91880700 1 -- start start 2024-01-18T06:54:57.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:57.951+0000 7faf91880700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf8c0f5410 0x7faf8c108920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:54:57.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:57.951+0000 7faf91880700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf8c108e10 con 0x7faf8c0f5410 2024-01-18T06:54:57.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:57.951+0000 7faf8affd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf8c0f5410 0x7faf8c108920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:54:57.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:57.951+0000 7faf8affd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf8c0f5410 0x7faf8c108920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:53430/0 (socket says 172.21.15.194:53430) 2024-01-18T06:54:57.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:57.951+0000 7faf8affd700 1 -- 172.21.15.194:0/2086015214 learned_addr learned my addr 172.21.15.194:0/2086015214 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:54:57.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:57.951+0000 7faf8affd700 1 -- 172.21.15.194:0/2086015214 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf80005040 con 0x7faf8c0f5410 2024-01-18T06:54:57.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:57.952+0000 7faf8affd700 1 --2- 172.21.15.194:0/2086015214 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf8c0f5410 0x7faf8c108920 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7faf80014000 tx=0x7faf8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:54:57.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:57.952+0000 7faf737fe700 1 -- 172.21.15.194:0/2086015214 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf800095f0 con 0x7faf8c0f5410 2024-01-18T06:54:57.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:57.952+0000 7faf91880700 1 -- 172.21.15.194:0/2086015214 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf8c109010 con 0x7faf8c0f5410 2024-01-18T06:54:57.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:57.952+0000 7faf91880700 1 -- 172.21.15.194:0/2086015214 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf8c1094a0 con 0x7faf8c0f5410 2024-01-18T06:54:57.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:57.952+0000 7faf737fe700 1 -- 172.21.15.194:0/2086015214 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faf80007e70 con 0x7faf8c0f5410 2024-01-18T06:54:57.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:57.952+0000 7faf737fe700 1 -- 172.21.15.194:0/2086015214 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf80014e50 con 0x7faf8c0f5410 2024-01-18T06:54:57.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:57.953+0000 7faf91880700 1 -- 172.21.15.194:0/2086015214 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf8c045ad0 con 0x7faf8c0f5410 2024-01-18T06:54:57.954 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:57.953+0000 7faf737fe700 1 -- 172.21.15.194:0/2086015214 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faf80025070 con 0x7faf8c0f5410 2024-01-18T06:54:57.954 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:57.953+0000 7faf737fe700 1 --2- 172.21.15.194:0/2086015214 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7faf7402ff10 0x7faf740323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:54:57.954 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:57.953+0000 7faf737fe700 1 -- 172.21.15.194:0/2086015214 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faf80040a00 con 0x7faf8c0f5410 2024-01-18T06:54:57.954 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:57.953+0000 7faf8a7fc700 1 --2- 172.21.15.194:0/2086015214 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7faf7402ff10 0x7faf740323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:54:57.954 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:57.954+0000 7faf8a7fc700 1 --2- 172.21.15.194:0/2086015214 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7faf7402ff10 0x7faf740323d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7faf7c0096a0 tx=0x7faf7c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:54:57.957 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:57.958+0000 7faf737fe700 1 -- 172.21.15.194:0/2086015214 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faf800079c0 con 0x7faf8c0f5410 2024-01-18T06:54:58.163 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:58.164+0000 7faf91880700 1 -- 172.21.15.194:0/2086015214 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faf8c10a2a0 con 0x7faf8c0f5410 2024-01-18T06:54:58.164 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:58.165+0000 7faf737fe700 1 -- 172.21.15.194:0/2086015214 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7faf8000f300 con 0x7faf8c0f5410 2024-01-18T06:54:58.164 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:54:58.164 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:54:58.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:58.166+0000 7faf91880700 1 -- 172.21.15.194:0/2086015214 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7faf7402ff10 msgr2=0x7faf740323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:58.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:58.166+0000 7faf91880700 1 --2- 172.21.15.194:0/2086015214 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7faf7402ff10 0x7faf740323d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7faf7c0096a0 tx=0x7faf7c006b40).stop 2024-01-18T06:54:58.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:58.166+0000 7faf91880700 1 -- 172.21.15.194:0/2086015214 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf8c0f5410 msgr2=0x7faf8c108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:54:58.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:58.166+0000 7faf91880700 1 --2- 172.21.15.194:0/2086015214 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf8c0f5410 0x7faf8c108920 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7faf80014000 tx=0x7faf8000b040).stop 2024-01-18T06:54:58.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:58.166+0000 7faf91880700 1 -- 172.21.15.194:0/2086015214 shutdown_connections 2024-01-18T06:54:58.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:58.166+0000 7faf91880700 1 --2- 172.21.15.194:0/2086015214 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7faf7402ff10 0x7faf740323d0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:58.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:58.166+0000 7faf91880700 1 --2- 172.21.15.194:0/2086015214 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf8c0f5410 0x7faf8c108920 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:54:58.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:58.166+0000 7faf91880700 1 -- 172.21.15.194:0/2086015214 >> 172.21.15.194:0/2086015214 conn(0x7faf8c0f0b70 msgr2=0x7faf8c05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:54:58.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:58.166+0000 7faf91880700 1 -- 172.21.15.194:0/2086015214 shutdown_connections 2024-01-18T06:54:58.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:54:58.166+0000 7faf91880700 1 -- 172.21.15.194:0/2086015214 wait complete. 2024-01-18T06:54:58.166 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:54:59.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:58 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:54:58.035567+0000 mgr.smithi047.fuhzzr (mgr.14162) 200 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:54:59.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:54:58 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:54:58.165549+0000 mon.smithi047 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.194:0/2086015214' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:54:59.550 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:54:59.551 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:55:00.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:00.767+0000 7f8a0f96d700 1 -- 172.21.15.194:0/3625590726 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a080f5410 msgr2=0x7f8a080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:00.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:00.767+0000 7f8a0f96d700 1 --2- 172.21.15.194:0/3625590726 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a080f5410 0x7f8a080f5830 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f8a00009960 tx=0x7f8a00004d10).stop 2024-01-18T06:55:00.768 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:00.767+0000 7f8a0f96d700 1 -- 172.21.15.194:0/3625590726 shutdown_connections 2024-01-18T06:55:00.768 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:00.767+0000 7f8a0f96d700 1 --2- 172.21.15.194:0/3625590726 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a080f5410 0x7f8a080f5830 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:00.768 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:00.767+0000 7f8a0f96d700 1 -- 172.21.15.194:0/3625590726 >> 172.21.15.194:0/3625590726 conn(0x7f8a080f0b70 msgr2=0x7f8a080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:00.768 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:00.767+0000 7f8a0f96d700 1 -- 172.21.15.194:0/3625590726 shutdown_connections 2024-01-18T06:55:00.768 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:00.767+0000 7f8a0f96d700 1 -- 172.21.15.194:0/3625590726 wait complete. 2024-01-18T06:55:00.768 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:00.768+0000 7f8a0f96d700 1 Processor -- start 2024-01-18T06:55:00.768 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:00.768+0000 7f8a0f96d700 1 -- start start 2024-01-18T06:55:00.768 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:00.768+0000 7f8a0f96d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a080f5410 0x7f8a08109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:00.768 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:00.768+0000 7f8a0f96d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a08060240 con 0x7f8a080f5410 2024-01-18T06:55:00.768 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:00.768+0000 7f8a0d709700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a080f5410 0x7f8a08109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:00.768 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:00.768+0000 7f8a0d709700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a080f5410 0x7f8a08109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:53444/0 (socket says 172.21.15.194:53444) 2024-01-18T06:55:00.768 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:00.768+0000 7f8a0d709700 1 -- 172.21.15.194:0/1238817349 learned_addr learned my addr 172.21.15.194:0/1238817349 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:55:00.770 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:00.768+0000 7f8a0d709700 1 -- 172.21.15.194:0/1238817349 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a00009640 con 0x7f8a080f5410 2024-01-18T06:55:00.770 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:00.769+0000 7f8a0d709700 1 --2- 172.21.15.194:0/1238817349 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a080f5410 0x7f8a08109430 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f8a00004b70 tx=0x7f8a00004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:00.770 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:00.769+0000 7f89fdffb700 1 -- 172.21.15.194:0/1238817349 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a00018070 con 0x7f8a080f5410 2024-01-18T06:55:00.770 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:00.769+0000 7f8a0f96d700 1 -- 172.21.15.194:0/1238817349 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a08060440 con 0x7f8a080f5410 2024-01-18T06:55:00.770 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:00.769+0000 7f89fdffb700 1 -- 172.21.15.194:0/1238817349 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8a0000b870 con 0x7f8a080f5410 2024-01-18T06:55:00.770 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:00.769+0000 7f89fdffb700 1 -- 172.21.15.194:0/1238817349 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a0001c560 con 0x7f8a080f5410 2024-01-18T06:55:00.770 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:00.769+0000 7f8a0f96d700 1 -- 172.21.15.194:0/1238817349 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a08109ae0 con 0x7f8a080f5410 2024-01-18T06:55:00.770 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:00.770+0000 7f89fdffb700 1 -- 172.21.15.194:0/1238817349 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8a00013630 con 0x7f8a080f5410 2024-01-18T06:55:00.771 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:00.770+0000 7f8a0f96d700 1 -- 172.21.15.194:0/1238817349 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a08045ad0 con 0x7f8a080f5410 2024-01-18T06:55:00.771 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:00.770+0000 7f89fdffb700 1 --2- 172.21.15.194:0/1238817349 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f89f402ff60 0x7f89f4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:00.771 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:00.770+0000 7f89fdffb700 1 -- 172.21.15.194:0/1238817349 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8a00041240 con 0x7f8a080f5410 2024-01-18T06:55:00.771 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:00.770+0000 7f8a0cf08700 1 --2- 172.21.15.194:0/1238817349 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f89f402ff60 0x7f89f4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:00.771 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:00.771+0000 7f8a0cf08700 1 --2- 172.21.15.194:0/1238817349 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f89f402ff60 0x7f89f4032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f89f80096a0 tx=0x7f89f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:00.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:00.775+0000 7f89fdffb700 1 -- 172.21.15.194:0/1238817349 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8a00004810 con 0x7f8a080f5410 2024-01-18T06:55:00.978 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:00.978+0000 7f8a0f96d700 1 -- 172.21.15.194:0/1238817349 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8a080fae60 con 0x7f8a080f5410 2024-01-18T06:55:00.978 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:00.979+0000 7f89fdffb700 1 -- 172.21.15.194:0/1238817349 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8a00012380 con 0x7f8a080f5410 2024-01-18T06:55:00.978 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:55:00.979 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:55:00.980 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:00.980+0000 7f8a0f96d700 1 -- 172.21.15.194:0/1238817349 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f89f402ff60 msgr2=0x7f89f4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:00.980 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:00.980+0000 7f8a0f96d700 1 --2- 172.21.15.194:0/1238817349 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f89f402ff60 0x7f89f4032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f89f80096a0 tx=0x7f89f8006b40).stop 2024-01-18T06:55:00.980 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:00.980+0000 7f8a0f96d700 1 -- 172.21.15.194:0/1238817349 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a080f5410 msgr2=0x7f8a08109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:00.980 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:00.980+0000 7f8a0f96d700 1 --2- 172.21.15.194:0/1238817349 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a080f5410 0x7f8a08109430 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f8a00004b70 tx=0x7f8a00004d10).stop 2024-01-18T06:55:00.980 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:00.980+0000 7f8a0f96d700 1 -- 172.21.15.194:0/1238817349 shutdown_connections 2024-01-18T06:55:00.980 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:00.980+0000 7f8a0f96d700 1 --2- 172.21.15.194:0/1238817349 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f89f402ff60 0x7f89f4032420 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:00.980 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:00.980+0000 7f8a0f96d700 1 --2- 172.21.15.194:0/1238817349 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8a080f5410 0x7f8a08109430 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:00.980 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:00.980+0000 7f8a0f96d700 1 -- 172.21.15.194:0/1238817349 >> 172.21.15.194:0/1238817349 conn(0x7f8a080f0b70 msgr2=0x7f8a080f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:00.981 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:00.980+0000 7f8a0f96d700 1 -- 172.21.15.194:0/1238817349 shutdown_connections 2024-01-18T06:55:00.981 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:00.980+0000 7f8a0f96d700 1 -- 172.21.15.194:0/1238817349 wait complete. 2024-01-18T06:55:00.981 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:55:01.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:01 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:55:00.035906+0000 mgr.smithi047.fuhzzr (mgr.14162) 201 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:55:01.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:01 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:55:00.979803+0000 mon.smithi047 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.194:0/1238817349' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:55:02.376 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:55:02.376 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:55:03.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:03 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:55:02.036177+0000 mgr.smithi047.fuhzzr (mgr.14162) 202 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:55:03.587 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:03.586+0000 7f54dd05a700 1 -- 172.21.15.194:0/536329994 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54d80f4860 msgr2=0x7f54d80f4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:03.587 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:03.586+0000 7f54dd05a700 1 --2- 172.21.15.194:0/536329994 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54d80f4860 0x7f54d80f4c80 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f54cc009fb0 tx=0x7f54cc009b30).stop 2024-01-18T06:55:03.587 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:03.586+0000 7f54dd05a700 1 -- 172.21.15.194:0/536329994 shutdown_connections 2024-01-18T06:55:03.588 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:03.586+0000 7f54dd05a700 1 --2- 172.21.15.194:0/536329994 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54d80f4860 0x7f54d80f4c80 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:03.588 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:03.586+0000 7f54dd05a700 1 -- 172.21.15.194:0/536329994 >> 172.21.15.194:0/536329994 conn(0x7f54d80f02e0 msgr2=0x7f54d80f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:03.588 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:03.586+0000 7f54dd05a700 1 -- 172.21.15.194:0/536329994 shutdown_connections 2024-01-18T06:55:03.588 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:03.586+0000 7f54dd05a700 1 -- 172.21.15.194:0/536329994 wait complete. 2024-01-18T06:55:03.588 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:03.587+0000 7f54dd05a700 1 Processor -- start 2024-01-18T06:55:03.588 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:03.587+0000 7f54dd05a700 1 -- start start 2024-01-18T06:55:03.588 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:03.587+0000 7f54dd05a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54d81069f0 0x7f54d8106e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:03.588 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:03.587+0000 7f54dd05a700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d8107300 con 0x7f54d81069f0 2024-01-18T06:55:03.588 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:03.587+0000 7f54d6d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54d81069f0 0x7f54d8106e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:03.588 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:03.587+0000 7f54d6d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54d81069f0 0x7f54d8106e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:53466/0 (socket says 172.21.15.194:53466) 2024-01-18T06:55:03.588 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:03.587+0000 7f54d6d9d700 1 -- 172.21.15.194:0/828066055 learned_addr learned my addr 172.21.15.194:0/828066055 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:55:03.588 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:03.588+0000 7f54d6d9d700 1 -- 172.21.15.194:0/828066055 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54cc005040 con 0x7f54d81069f0 2024-01-18T06:55:03.588 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:03.588+0000 7f54d6d9d700 1 --2- 172.21.15.194:0/828066055 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54d81069f0 0x7f54d8106e10 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f54cc009f80 tx=0x7f54cc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:03.589 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:03.588+0000 7f54bf7fe700 1 -- 172.21.15.194:0/828066055 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54cc0136c0 con 0x7f54d81069f0 2024-01-18T06:55:03.589 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:03.588+0000 7f54dd05a700 1 -- 172.21.15.194:0/828066055 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54d8107500 con 0x7f54d81069f0 2024-01-18T06:55:03.589 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:03.588+0000 7f54bf7fe700 1 -- 172.21.15.194:0/828066055 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f54cc0095f0 con 0x7f54d81069f0 2024-01-18T06:55:03.589 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:03.588+0000 7f54bf7fe700 1 -- 172.21.15.194:0/828066055 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54cc014420 con 0x7f54d81069f0 2024-01-18T06:55:03.589 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:03.588+0000 7f54dd05a700 1 -- 172.21.15.194:0/828066055 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54d8108050 con 0x7f54d81069f0 2024-01-18T06:55:03.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:03.589+0000 7f54dd05a700 1 -- 172.21.15.194:0/828066055 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54d8059100 con 0x7f54d81069f0 2024-01-18T06:55:03.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:03.589+0000 7f54bf7fe700 1 -- 172.21.15.194:0/828066055 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f54cc025060 con 0x7f54d81069f0 2024-01-18T06:55:03.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:03.589+0000 7f54bf7fe700 1 --2- 172.21.15.194:0/828066055 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f54c002ff60 0x7f54c0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:03.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:03.589+0000 7f54bf7fe700 1 -- 172.21.15.194:0/828066055 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f54cc040e90 con 0x7f54d81069f0 2024-01-18T06:55:03.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:03.589+0000 7f54d659c700 1 --2- 172.21.15.194:0/828066055 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f54c002ff60 0x7f54c0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:03.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:03.590+0000 7f54d659c700 1 --2- 172.21.15.194:0/828066055 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f54c002ff60 0x7f54c0032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f54c80096a0 tx=0x7f54c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:03.593 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:03.594+0000 7f54bf7fe700 1 -- 172.21.15.194:0/828066055 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f54cc004490 con 0x7f54d81069f0 2024-01-18T06:55:03.799 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:03.799+0000 7f54dd05a700 1 -- 172.21.15.194:0/828066055 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f54d8045ad0 con 0x7f54d81069f0 2024-01-18T06:55:03.799 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:03.800+0000 7f54bf7fe700 1 -- 172.21.15.194:0/828066055 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f54cc004490 con 0x7f54d81069f0 2024-01-18T06:55:03.800 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:55:03.800 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:55:03.801 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:03.802+0000 7f54dd05a700 1 -- 172.21.15.194:0/828066055 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f54c002ff60 msgr2=0x7f54c0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:03.801 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:03.802+0000 7f54dd05a700 1 --2- 172.21.15.194:0/828066055 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f54c002ff60 0x7f54c0032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f54c80096a0 tx=0x7f54c8006b40).stop 2024-01-18T06:55:03.801 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:03.802+0000 7f54dd05a700 1 -- 172.21.15.194:0/828066055 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54d81069f0 msgr2=0x7f54d8106e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:03.801 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:03.802+0000 7f54dd05a700 1 --2- 172.21.15.194:0/828066055 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54d81069f0 0x7f54d8106e10 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f54cc009f80 tx=0x7f54cc009b30).stop 2024-01-18T06:55:03.801 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:03.802+0000 7f54dd05a700 1 -- 172.21.15.194:0/828066055 shutdown_connections 2024-01-18T06:55:03.802 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:03.802+0000 7f54dd05a700 1 --2- 172.21.15.194:0/828066055 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f54c002ff60 0x7f54c0032420 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:03.802 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:03.802+0000 7f54dd05a700 1 --2- 172.21.15.194:0/828066055 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54d81069f0 0x7f54d8106e10 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:03.802 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:03.802+0000 7f54dd05a700 1 -- 172.21.15.194:0/828066055 >> 172.21.15.194:0/828066055 conn(0x7f54d80f02e0 msgr2=0x7f54d80f0e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:03.802 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:03.802+0000 7f54dd05a700 1 -- 172.21.15.194:0/828066055 shutdown_connections 2024-01-18T06:55:03.802 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:03.802+0000 7f54dd05a700 1 -- 172.21.15.194:0/828066055 wait complete. 2024-01-18T06:55:03.802 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:55:04.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:04 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:55:03.801072+0000 mon.smithi047 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.194:0/828066055' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:55:05.186 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:55:05.186 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:55:05.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:05 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:55:04.036551+0000 mgr.smithi047.fuhzzr (mgr.14162) 203 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:55:06.340 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:06.339+0000 7f5635644700 1 -- 172.21.15.194:0/2892571986 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f56300f5410 msgr2=0x7f56300f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:06.340 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:06.339+0000 7f5635644700 1 --2- 172.21.15.194:0/2892571986 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f56300f5410 0x7f56300f5830 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f5624009fb0 tx=0x7f5624009b30).stop 2024-01-18T06:55:06.340 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:06.339+0000 7f5635644700 1 -- 172.21.15.194:0/2892571986 shutdown_connections 2024-01-18T06:55:06.340 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:06.339+0000 7f5635644700 1 --2- 172.21.15.194:0/2892571986 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f56300f5410 0x7f56300f5830 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:06.340 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:06.339+0000 7f5635644700 1 -- 172.21.15.194:0/2892571986 >> 172.21.15.194:0/2892571986 conn(0x7f56300f0b70 msgr2=0x7f56300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:06.340 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:06.339+0000 7f5635644700 1 -- 172.21.15.194:0/2892571986 shutdown_connections 2024-01-18T06:55:06.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:06.339+0000 7f5635644700 1 -- 172.21.15.194:0/2892571986 wait complete. 2024-01-18T06:55:06.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:06.339+0000 7f5635644700 1 Processor -- start 2024-01-18T06:55:06.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:06.339+0000 7f5635644700 1 -- start start 2024-01-18T06:55:06.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:06.340+0000 7f5635644700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f56300f5410 0x7f5630109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:06.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:06.340+0000 7f5635644700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5630060240 con 0x7f56300f5410 2024-01-18T06:55:06.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:06.340+0000 7f562effd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f56300f5410 0x7f5630109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:06.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:06.340+0000 7f562effd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f56300f5410 0x7f5630109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:53486/0 (socket says 172.21.15.194:53486) 2024-01-18T06:55:06.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:06.340+0000 7f562effd700 1 -- 172.21.15.194:0/11904077 learned_addr learned my addr 172.21.15.194:0/11904077 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:55:06.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:06.340+0000 7f562effd700 1 -- 172.21.15.194:0/11904077 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5624005040 con 0x7f56300f5410 2024-01-18T06:55:06.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:06.340+0000 7f562effd700 1 --2- 172.21.15.194:0/11904077 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f56300f5410 0x7f5630109400 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f5624013510 tx=0x7f562400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:06.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:06.340+0000 7f56177fe700 1 -- 172.21.15.194:0/11904077 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5624009320 con 0x7f56300f5410 2024-01-18T06:55:06.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:06.341+0000 7f5635644700 1 -- 172.21.15.194:0/11904077 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5630060440 con 0x7f56300f5410 2024-01-18T06:55:06.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:06.341+0000 7f56177fe700 1 -- 172.21.15.194:0/11904077 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5624013f00 con 0x7f56300f5410 2024-01-18T06:55:06.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:06.341+0000 7f56177fe700 1 -- 172.21.15.194:0/11904077 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5624014d90 con 0x7f56300f5410 2024-01-18T06:55:06.342 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:06.341+0000 7f5635644700 1 -- 172.21.15.194:0/11904077 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5630109ab0 con 0x7f56300f5410 2024-01-18T06:55:06.342 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:06.341+0000 7f56177fe700 1 -- 172.21.15.194:0/11904077 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5624014420 con 0x7f56300f5410 2024-01-18T06:55:06.343 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:06.341+0000 7f5635644700 1 -- 172.21.15.194:0/11904077 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5630045ad0 con 0x7f56300f5410 2024-01-18T06:55:06.343 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:06.341+0000 7f56177fe700 1 --2- 172.21.15.194:0/11904077 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f561802ff60 0x7f5618032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:06.343 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:06.342+0000 7f56177fe700 1 -- 172.21.15.194:0/11904077 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f56240410c0 con 0x7f56300f5410 2024-01-18T06:55:06.343 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:06.342+0000 7f562e7fc700 1 --2- 172.21.15.194:0/11904077 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f561802ff60 0x7f5618032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:06.343 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:06.342+0000 7f562e7fc700 1 --2- 172.21.15.194:0/11904077 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f561802ff60 0x7f5618032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f56200096a0 tx=0x7f5620006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:06.346 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:06.346+0000 7f56177fe700 1 -- 172.21.15.194:0/11904077 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f56240095f0 con 0x7f56300f5410 2024-01-18T06:55:06.553 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:06.554+0000 7f5635644700 1 -- 172.21.15.194:0/11904077 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f56300fae60 con 0x7f56300f5410 2024-01-18T06:55:06.554 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:06.555+0000 7f56177fe700 1 -- 172.21.15.194:0/11904077 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f562401db70 con 0x7f56300f5410 2024-01-18T06:55:06.554 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:55:06.554 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:55:06.556 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:06.556+0000 7f5635644700 1 -- 172.21.15.194:0/11904077 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f561802ff60 msgr2=0x7f5618032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:06.556 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:06.556+0000 7f5635644700 1 --2- 172.21.15.194:0/11904077 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f561802ff60 0x7f5618032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f56200096a0 tx=0x7f5620006b40).stop 2024-01-18T06:55:06.556 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:06.556+0000 7f5635644700 1 -- 172.21.15.194:0/11904077 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f56300f5410 msgr2=0x7f5630109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:06.556 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:06.556+0000 7f5635644700 1 --2- 172.21.15.194:0/11904077 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f56300f5410 0x7f5630109400 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f5624013510 tx=0x7f562400b040).stop 2024-01-18T06:55:06.556 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:06.556+0000 7f5635644700 1 -- 172.21.15.194:0/11904077 shutdown_connections 2024-01-18T06:55:06.556 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:06.556+0000 7f5635644700 1 --2- 172.21.15.194:0/11904077 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f561802ff60 0x7f5618032420 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:06.556 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:06.556+0000 7f5635644700 1 --2- 172.21.15.194:0/11904077 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f56300f5410 0x7f5630109400 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:06.556 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:06.556+0000 7f5635644700 1 -- 172.21.15.194:0/11904077 >> 172.21.15.194:0/11904077 conn(0x7f56300f0b70 msgr2=0x7f56300f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:06.556 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:06.556+0000 7f5635644700 1 -- 172.21.15.194:0/11904077 shutdown_connections 2024-01-18T06:55:06.557 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:06.556+0000 7f5635644700 1 -- 172.21.15.194:0/11904077 wait complete. 2024-01-18T06:55:06.557 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:55:07.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:07 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:55:06.036893+0000 mgr.smithi047.fuhzzr (mgr.14162) 204 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:55:07.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:07 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:55:06.555615+0000 mon.smithi047 (mon.0) 268 : audit [DBG] from='client.? 172.21.15.194:0/11904077' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:55:07.932 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:55:07.932 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:55:09.150 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:09.149+0000 7fc16005b700 1 -- 172.21.15.194:0/3986130468 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1580f6480 msgr2=0x7fc1580f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:09.151 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:09.149+0000 7fc16005b700 1 --2- 172.21.15.194:0/3986130468 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1580f6480 0x7fc1580f68a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fc14c009fb0 tx=0x7fc14c009b30).stop 2024-01-18T06:55:09.151 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:09.149+0000 7fc16005b700 1 -- 172.21.15.194:0/3986130468 shutdown_connections 2024-01-18T06:55:09.151 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:09.149+0000 7fc16005b700 1 --2- 172.21.15.194:0/3986130468 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1580f6480 0x7fc1580f68a0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:09.151 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:09.149+0000 7fc16005b700 1 -- 172.21.15.194:0/3986130468 >> 172.21.15.194:0/3986130468 conn(0x7fc1580f1c00 msgr2=0x7fc1580f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:09.151 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:09.149+0000 7fc16005b700 1 -- 172.21.15.194:0/3986130468 shutdown_connections 2024-01-18T06:55:09.151 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:09.149+0000 7fc16005b700 1 -- 172.21.15.194:0/3986130468 wait complete. 2024-01-18T06:55:09.151 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:09.150+0000 7fc16005b700 1 Processor -- start 2024-01-18T06:55:09.151 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:09.150+0000 7fc16005b700 1 -- start start 2024-01-18T06:55:09.151 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:09.150+0000 7fc16005b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1580f6480 0x7fc1581066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:09.152 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:09.150+0000 7fc16005b700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc158106bc0 con 0x7fc1580f6480 2024-01-18T06:55:09.152 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:09.150+0000 7fc15ddf7700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1580f6480 0x7fc1581066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:09.152 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:09.150+0000 7fc15ddf7700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1580f6480 0x7fc1581066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:47324/0 (socket says 172.21.15.194:47324) 2024-01-18T06:55:09.152 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:09.150+0000 7fc15ddf7700 1 -- 172.21.15.194:0/3306557899 learned_addr learned my addr 172.21.15.194:0/3306557899 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:55:09.152 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:09.150+0000 7fc15ddf7700 1 -- 172.21.15.194:0/3306557899 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc14c005040 con 0x7fc1580f6480 2024-01-18T06:55:09.152 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:09.151+0000 7fc15ddf7700 1 --2- 172.21.15.194:0/3306557899 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1580f6480 0x7fc1581066d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fc14c014000 tx=0x7fc14c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:09.152 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:09.151+0000 7fc14a7fc700 1 -- 172.21.15.194:0/3306557899 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc14c0095f0 con 0x7fc1580f6480 2024-01-18T06:55:09.152 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:09.151+0000 7fc16005b700 1 -- 172.21.15.194:0/3306557899 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc158106dc0 con 0x7fc1580f6480 2024-01-18T06:55:09.152 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:09.151+0000 7fc14a7fc700 1 -- 172.21.15.194:0/3306557899 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc14c007e70 con 0x7fc1580f6480 2024-01-18T06:55:09.153 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:09.151+0000 7fc16005b700 1 -- 172.21.15.194:0/3306557899 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc158107250 con 0x7fc1580f6480 2024-01-18T06:55:09.153 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:09.151+0000 7fc14a7fc700 1 -- 172.21.15.194:0/3306557899 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc14c014ea0 con 0x7fc1580f6480 2024-01-18T06:55:09.154 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:09.152+0000 7fc14a7fc700 1 -- 172.21.15.194:0/3306557899 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc14c025070 con 0x7fc1580f6480 2024-01-18T06:55:09.154 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:09.152+0000 7fc16005b700 1 -- 172.21.15.194:0/3306557899 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc158045ad0 con 0x7fc1580f6480 2024-01-18T06:55:09.154 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:09.152+0000 7fc14a7fc700 1 --2- 172.21.15.194:0/3306557899 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc14402ff60 0x7fc144032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:09.154 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:09.152+0000 7fc14a7fc700 1 -- 172.21.15.194:0/3306557899 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc14c040930 con 0x7fc1580f6480 2024-01-18T06:55:09.154 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:09.152+0000 7fc15d5f6700 1 --2- 172.21.15.194:0/3306557899 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc14402ff60 0x7fc144032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:09.154 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:09.152+0000 7fc15d5f6700 1 --2- 172.21.15.194:0/3306557899 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc14402ff60 0x7fc144032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fc1540096a0 tx=0x7fc154006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:09.156 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:09.157+0000 7fc14a7fc700 1 -- 172.21.15.194:0/3306557899 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc14c0079c0 con 0x7fc1580f6480 2024-01-18T06:55:09.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:09.365+0000 7fc16005b700 1 -- 172.21.15.194:0/3306557899 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc1581080a0 con 0x7fc1580f6480 2024-01-18T06:55:09.365 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:09.366+0000 7fc14a7fc700 1 -- 172.21.15.194:0/3306557899 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc14c00f300 con 0x7fc1580f6480 2024-01-18T06:55:09.365 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:55:09.365 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:55:09.367 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:09.367+0000 7fc16005b700 1 -- 172.21.15.194:0/3306557899 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc14402ff60 msgr2=0x7fc144032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:09.367 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:09.367+0000 7fc16005b700 1 --2- 172.21.15.194:0/3306557899 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc14402ff60 0x7fc144032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fc1540096a0 tx=0x7fc154006b40).stop 2024-01-18T06:55:09.367 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:09.367+0000 7fc16005b700 1 -- 172.21.15.194:0/3306557899 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1580f6480 msgr2=0x7fc1581066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:09.367 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:09.367+0000 7fc16005b700 1 --2- 172.21.15.194:0/3306557899 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1580f6480 0x7fc1581066d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fc14c014000 tx=0x7fc14c00b040).stop 2024-01-18T06:55:09.367 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:09.368+0000 7fc16005b700 1 -- 172.21.15.194:0/3306557899 shutdown_connections 2024-01-18T06:55:09.367 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:09.368+0000 7fc16005b700 1 --2- 172.21.15.194:0/3306557899 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc14402ff60 0x7fc144032420 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:09.367 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:09.368+0000 7fc16005b700 1 --2- 172.21.15.194:0/3306557899 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc1580f6480 0x7fc1581066d0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:09.367 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:09.368+0000 7fc16005b700 1 -- 172.21.15.194:0/3306557899 >> 172.21.15.194:0/3306557899 conn(0x7fc1580f1c00 msgr2=0x7fc1580f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:09.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:09.368+0000 7fc16005b700 1 -- 172.21.15.194:0/3306557899 shutdown_connections 2024-01-18T06:55:09.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:09.368+0000 7fc16005b700 1 -- 172.21.15.194:0/3306557899 wait complete. 2024-01-18T06:55:09.368 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:55:09.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:08 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:55:08.037236+0000 mgr.smithi047.fuhzzr (mgr.14162) 205 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:55:10.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:09 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:55:09.366753+0000 mon.smithi047 (mon.0) 269 : audit [DBG] from='client.? 172.21.15.194:0/3306557899' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:55:10.749 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:55:10.750 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:55:11.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:10 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:55:10.037650+0000 mgr.smithi047.fuhzzr (mgr.14162) 206 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:55:11.946 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:11.946+0000 7f75f5994700 1 -- 172.21.15.194:0/3362600502 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75f00f5410 msgr2=0x7f75f00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:11.946 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:11.946+0000 7f75f5994700 1 --2- 172.21.15.194:0/3362600502 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75f00f5410 0x7f75f00f5830 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f75d8009fb0 tx=0x7f75d8009b30).stop 2024-01-18T06:55:11.946 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:11.946+0000 7f75f5994700 1 -- 172.21.15.194:0/3362600502 shutdown_connections 2024-01-18T06:55:11.946 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:11.946+0000 7f75f5994700 1 --2- 172.21.15.194:0/3362600502 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75f00f5410 0x7f75f00f5830 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:11.946 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:11.946+0000 7f75f5994700 1 -- 172.21.15.194:0/3362600502 >> 172.21.15.194:0/3362600502 conn(0x7f75f00f0b70 msgr2=0x7f75f00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:11.946 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:11.946+0000 7f75f5994700 1 -- 172.21.15.194:0/3362600502 shutdown_connections 2024-01-18T06:55:11.946 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:11.946+0000 7f75f5994700 1 -- 172.21.15.194:0/3362600502 wait complete. 2024-01-18T06:55:11.948 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:11.947+0000 7f75f5994700 1 Processor -- start 2024-01-18T06:55:11.948 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:11.947+0000 7f75f5994700 1 -- start start 2024-01-18T06:55:11.949 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:11.947+0000 7f75f5994700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75f00f5410 0x7f75f01093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:11.949 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:11.947+0000 7f75f5994700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f0060220 con 0x7f75f00f5410 2024-01-18T06:55:11.949 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:11.947+0000 7f75eeffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75f00f5410 0x7f75f01093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:11.949 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:11.947+0000 7f75eeffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75f00f5410 0x7f75f01093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:47346/0 (socket says 172.21.15.194:47346) 2024-01-18T06:55:11.949 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:11.947+0000 7f75eeffd700 1 -- 172.21.15.194:0/318720864 learned_addr learned my addr 172.21.15.194:0/318720864 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:55:11.949 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:11.948+0000 7f75eeffd700 1 -- 172.21.15.194:0/318720864 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d8005040 con 0x7f75f00f5410 2024-01-18T06:55:11.949 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:11.948+0000 7f75eeffd700 1 --2- 172.21.15.194:0/318720864 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75f00f5410 0x7f75f01093f0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f75d8013510 tx=0x7f75d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:11.949 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:11.948+0000 7f75e77fe700 1 -- 172.21.15.194:0/318720864 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75d8009320 con 0x7f75f00f5410 2024-01-18T06:55:11.949 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:11.948+0000 7f75e77fe700 1 -- 172.21.15.194:0/318720864 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f75d8013f00 con 0x7f75f00f5410 2024-01-18T06:55:11.949 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:11.948+0000 7f75f5994700 1 -- 172.21.15.194:0/318720864 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75f0060420 con 0x7f75f00f5410 2024-01-18T06:55:11.949 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:11.948+0000 7f75e77fe700 1 -- 172.21.15.194:0/318720864 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75d8014d90 con 0x7f75f00f5410 2024-01-18T06:55:11.949 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:11.948+0000 7f75f5994700 1 -- 172.21.15.194:0/318720864 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75f0109aa0 con 0x7f75f00f5410 2024-01-18T06:55:11.949 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:11.949+0000 7f75e77fe700 1 -- 172.21.15.194:0/318720864 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f75d80095f0 con 0x7f75f00f5410 2024-01-18T06:55:11.950 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:11.949+0000 7f75f5994700 1 -- 172.21.15.194:0/318720864 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75f0059100 con 0x7f75f00f5410 2024-01-18T06:55:11.950 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:11.949+0000 7f75e77fe700 1 --2- 172.21.15.194:0/318720864 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f75dc02ffb0 0x7f75dc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:11.950 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:11.949+0000 7f75e77fe700 1 -- 172.21.15.194:0/318720864 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f75d8040f90 con 0x7f75f00f5410 2024-01-18T06:55:11.950 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:11.949+0000 7f75ee7fc700 1 --2- 172.21.15.194:0/318720864 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f75dc02ffb0 0x7f75dc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:11.950 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:11.950+0000 7f75ee7fc700 1 --2- 172.21.15.194:0/318720864 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f75dc02ffb0 0x7f75dc032470 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f75e00096a0 tx=0x7f75e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:11.954 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:11.954+0000 7f75e77fe700 1 -- 172.21.15.194:0/318720864 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f75d8014420 con 0x7f75f00f5410 2024-01-18T06:55:12.157 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:12.158+0000 7f75f5994700 1 -- 172.21.15.194:0/318720864 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f75f0045ad0 con 0x7f75f00f5410 2024-01-18T06:55:12.158 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:12.159+0000 7f75e77fe700 1 -- 172.21.15.194:0/318720864 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f75d801d7f0 con 0x7f75f00f5410 2024-01-18T06:55:12.158 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:55:12.158 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:55:12.160 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:12.160+0000 7f75f5994700 1 -- 172.21.15.194:0/318720864 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f75dc02ffb0 msgr2=0x7f75dc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:12.160 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:12.160+0000 7f75f5994700 1 --2- 172.21.15.194:0/318720864 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f75dc02ffb0 0x7f75dc032470 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f75e00096a0 tx=0x7f75e0006b40).stop 2024-01-18T06:55:12.160 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:12.160+0000 7f75f5994700 1 -- 172.21.15.194:0/318720864 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75f00f5410 msgr2=0x7f75f01093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:12.160 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:12.160+0000 7f75f5994700 1 --2- 172.21.15.194:0/318720864 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75f00f5410 0x7f75f01093f0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f75d8013510 tx=0x7f75d800b040).stop 2024-01-18T06:55:12.160 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:12.160+0000 7f75f5994700 1 -- 172.21.15.194:0/318720864 shutdown_connections 2024-01-18T06:55:12.160 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:12.160+0000 7f75f5994700 1 --2- 172.21.15.194:0/318720864 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f75dc02ffb0 0x7f75dc032470 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:12.160 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:12.160+0000 7f75f5994700 1 --2- 172.21.15.194:0/318720864 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75f00f5410 0x7f75f01093f0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:12.160 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:12.160+0000 7f75f5994700 1 -- 172.21.15.194:0/318720864 >> 172.21.15.194:0/318720864 conn(0x7f75f00f0b70 msgr2=0x7f75f00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:12.160 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:12.160+0000 7f75f5994700 1 -- 172.21.15.194:0/318720864 shutdown_connections 2024-01-18T06:55:12.161 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:12.160+0000 7f75f5994700 1 -- 172.21.15.194:0/318720864 wait complete. 2024-01-18T06:55:12.161 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:55:13.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:13 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:55:12.037937+0000 mgr.smithi047.fuhzzr (mgr.14162) 207 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:55:13.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:13 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:55:12.159629+0000 mon.smithi047 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.194:0/318720864' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:55:13.539 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:55:13.539 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:55:14.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:14.679+0000 7f3d26a69700 1 -- 172.21.15.194:0/2823207551 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d200f51a0 msgr2=0x7f3d200f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:14.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:14.679+0000 7f3d26a69700 1 --2- 172.21.15.194:0/2823207551 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d200f51a0 0x7f3d200f55c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f3d1c009fb0 tx=0x7f3d1c009b30).stop 2024-01-18T06:55:14.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:14.679+0000 7f3d26a69700 1 -- 172.21.15.194:0/2823207551 shutdown_connections 2024-01-18T06:55:14.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:14.679+0000 7f3d26a69700 1 --2- 172.21.15.194:0/2823207551 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d200f51a0 0x7f3d200f55c0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:14.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:14.679+0000 7f3d26a69700 1 -- 172.21.15.194:0/2823207551 >> 172.21.15.194:0/2823207551 conn(0x7f3d200f09a0 msgr2=0x7f3d200f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:14.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:14.679+0000 7f3d26a69700 1 -- 172.21.15.194:0/2823207551 shutdown_connections 2024-01-18T06:55:14.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:14.679+0000 7f3d26a69700 1 -- 172.21.15.194:0/2823207551 wait complete. 2024-01-18T06:55:14.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:14.680+0000 7f3d26a69700 1 Processor -- start 2024-01-18T06:55:14.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:14.680+0000 7f3d26a69700 1 -- start start 2024-01-18T06:55:14.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:14.680+0000 7f3d26a69700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d200f51a0 0x7f3d201064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:14.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:14.680+0000 7f3d26a69700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d201069d0 con 0x7f3d200f51a0 2024-01-18T06:55:14.682 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:14.680+0000 7f3d25a67700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d200f51a0 0x7f3d201064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:14.682 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:14.680+0000 7f3d25a67700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d200f51a0 0x7f3d201064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:47364/0 (socket says 172.21.15.194:47364) 2024-01-18T06:55:14.682 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:14.680+0000 7f3d25a67700 1 -- 172.21.15.194:0/3619787837 learned_addr learned my addr 172.21.15.194:0/3619787837 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:55:14.682 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:14.680+0000 7f3d25a67700 1 -- 172.21.15.194:0/3619787837 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d1c005040 con 0x7f3d200f51a0 2024-01-18T06:55:14.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:14.680+0000 7f3d25a67700 1 --2- 172.21.15.194:0/3619787837 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d200f51a0 0x7f3d201064e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f3d1c013510 tx=0x7f3d1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:14.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:14.681+0000 7f3d167fc700 1 -- 172.21.15.194:0/3619787837 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d1c009320 con 0x7f3d200f51a0 2024-01-18T06:55:14.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:14.681+0000 7f3d26a69700 1 -- 172.21.15.194:0/3619787837 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d20106bd0 con 0x7f3d200f51a0 2024-01-18T06:55:14.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:14.681+0000 7f3d167fc700 1 -- 172.21.15.194:0/3619787837 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3d1c013f00 con 0x7f3d200f51a0 2024-01-18T06:55:14.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:14.681+0000 7f3d167fc700 1 -- 172.21.15.194:0/3619787837 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d1c014dc0 con 0x7f3d200f51a0 2024-01-18T06:55:14.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:14.681+0000 7f3d26a69700 1 -- 172.21.15.194:0/3619787837 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d20107060 con 0x7f3d200f51a0 2024-01-18T06:55:14.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:14.682+0000 7f3d26a69700 1 -- 172.21.15.194:0/3619787837 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d20045ad0 con 0x7f3d200f51a0 2024-01-18T06:55:14.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:14.682+0000 7f3d167fc700 1 -- 172.21.15.194:0/3619787837 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3d1c0095f0 con 0x7f3d200f51a0 2024-01-18T06:55:14.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:14.682+0000 7f3d167fc700 1 --2- 172.21.15.194:0/3619787837 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3d0c02ff60 0x7f3d0c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:14.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:14.682+0000 7f3d167fc700 1 -- 172.21.15.194:0/3619787837 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3d1c041000 con 0x7f3d200f51a0 2024-01-18T06:55:14.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:14.682+0000 7f3d25266700 1 --2- 172.21.15.194:0/3619787837 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3d0c02ff60 0x7f3d0c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:14.684 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:14.682+0000 7f3d25266700 1 --2- 172.21.15.194:0/3619787837 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3d0c02ff60 0x7f3d0c032420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f3d100096a0 tx=0x7f3d10006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:14.686 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:14.687+0000 7f3d167fc700 1 -- 172.21.15.194:0/3619787837 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3d1c01c790 con 0x7f3d200f51a0 2024-01-18T06:55:14.896 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:14.896+0000 7f3d26a69700 1 -- 172.21.15.194:0/3619787837 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3d20107f10 con 0x7f3d200f51a0 2024-01-18T06:55:14.896 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:14.897+0000 7f3d167fc700 1 -- 172.21.15.194:0/3619787837 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3d1c010230 con 0x7f3d200f51a0 2024-01-18T06:55:14.897 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:55:14.897 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:55:14.898 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:14.898+0000 7f3d26a69700 1 -- 172.21.15.194:0/3619787837 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3d0c02ff60 msgr2=0x7f3d0c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:14.898 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:14.898+0000 7f3d26a69700 1 --2- 172.21.15.194:0/3619787837 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3d0c02ff60 0x7f3d0c032420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f3d100096a0 tx=0x7f3d10006b40).stop 2024-01-18T06:55:14.898 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:14.898+0000 7f3d26a69700 1 -- 172.21.15.194:0/3619787837 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d200f51a0 msgr2=0x7f3d201064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:14.898 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:14.898+0000 7f3d26a69700 1 --2- 172.21.15.194:0/3619787837 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d200f51a0 0x7f3d201064e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f3d1c013510 tx=0x7f3d1c00b040).stop 2024-01-18T06:55:14.898 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:14.899+0000 7f3d26a69700 1 -- 172.21.15.194:0/3619787837 shutdown_connections 2024-01-18T06:55:14.898 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:14.899+0000 7f3d26a69700 1 --2- 172.21.15.194:0/3619787837 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3d0c02ff60 0x7f3d0c032420 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:14.898 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:14.899+0000 7f3d26a69700 1 --2- 172.21.15.194:0/3619787837 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3d200f51a0 0x7f3d201064e0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:14.898 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:14.899+0000 7f3d26a69700 1 -- 172.21.15.194:0/3619787837 >> 172.21.15.194:0/3619787837 conn(0x7f3d200f09a0 msgr2=0x7f3d2005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:14.898 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:14.899+0000 7f3d26a69700 1 -- 172.21.15.194:0/3619787837 shutdown_connections 2024-01-18T06:55:14.899 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:14.899+0000 7f3d26a69700 1 -- 172.21.15.194:0/3619787837 wait complete. 2024-01-18T06:55:14.899 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:55:15.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:15 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:55:14.038262+0000 mgr.smithi047.fuhzzr (mgr.14162) 208 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:55:15.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:15 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:55:14.897998+0000 mon.smithi047 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.194:0/3619787837' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:55:16.290 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:55:16.291 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:55:17.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:17 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:55:16.038651+0000 mgr.smithi047.fuhzzr (mgr.14162) 209 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:55:17.539 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:17.538+0000 7face764f700 1 -- 172.21.15.194:0/2958201748 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7face00f5410 msgr2=0x7face00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:17.539 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:17.538+0000 7face764f700 1 --2- 172.21.15.194:0/2958201748 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7face00f5410 0x7face00f5830 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7facd0009fb0 tx=0x7facd0009b30).stop 2024-01-18T06:55:17.539 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:17.539+0000 7face764f700 1 -- 172.21.15.194:0/2958201748 shutdown_connections 2024-01-18T06:55:17.539 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:17.539+0000 7face764f700 1 --2- 172.21.15.194:0/2958201748 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7face00f5410 0x7face00f5830 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:17.539 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:17.539+0000 7face764f700 1 -- 172.21.15.194:0/2958201748 >> 172.21.15.194:0/2958201748 conn(0x7face00f0b70 msgr2=0x7face00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:17.539 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:17.539+0000 7face764f700 1 -- 172.21.15.194:0/2958201748 shutdown_connections 2024-01-18T06:55:17.539 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:17.539+0000 7face764f700 1 -- 172.21.15.194:0/2958201748 wait complete. 2024-01-18T06:55:17.539 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:17.539+0000 7face764f700 1 Processor -- start 2024-01-18T06:55:17.540 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:17.539+0000 7face764f700 1 -- start start 2024-01-18T06:55:17.540 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:17.539+0000 7face764f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7face00f5410 0x7face0109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:17.540 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:17.539+0000 7face764f700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face0060240 con 0x7face00f5410 2024-01-18T06:55:17.540 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:17.540+0000 7face53eb700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7face00f5410 0x7face0109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:17.540 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:17.540+0000 7face53eb700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7face00f5410 0x7face0109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:39410/0 (socket says 172.21.15.194:39410) 2024-01-18T06:55:17.540 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:17.540+0000 7face53eb700 1 -- 172.21.15.194:0/3444116987 learned_addr learned my addr 172.21.15.194:0/3444116987 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:55:17.541 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:17.540+0000 7face53eb700 1 -- 172.21.15.194:0/3444116987 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd0005040 con 0x7face00f5410 2024-01-18T06:55:17.541 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:17.540+0000 7face53eb700 1 --2- 172.21.15.194:0/3444116987 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7face00f5410 0x7face0109430 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7facd0014000 tx=0x7facd000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:17.542 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:17.540+0000 7facd5ffb700 1 -- 172.21.15.194:0/3444116987 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7facd00095f0 con 0x7face00f5410 2024-01-18T06:55:17.542 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:17.540+0000 7facd5ffb700 1 -- 172.21.15.194:0/3444116987 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7facd0007e70 con 0x7face00f5410 2024-01-18T06:55:17.542 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:17.540+0000 7face764f700 1 -- 172.21.15.194:0/3444116987 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7face0060440 con 0x7face00f5410 2024-01-18T06:55:17.542 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:17.540+0000 7facd5ffb700 1 -- 172.21.15.194:0/3444116987 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7facd0014ea0 con 0x7face00f5410 2024-01-18T06:55:17.542 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:17.541+0000 7face764f700 1 -- 172.21.15.194:0/3444116987 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7face0109ae0 con 0x7face00f5410 2024-01-18T06:55:17.542 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:17.541+0000 7face764f700 1 -- 172.21.15.194:0/3444116987 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7face0045ad0 con 0x7face00f5410 2024-01-18T06:55:17.542 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:17.541+0000 7facd5ffb700 1 -- 172.21.15.194:0/3444116987 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7facd0025070 con 0x7face00f5410 2024-01-18T06:55:17.542 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:17.542+0000 7facd5ffb700 1 --2- 172.21.15.194:0/3444116987 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7faccc02ffb0 0x7faccc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:17.542 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:17.542+0000 7facd5ffb700 1 -- 172.21.15.194:0/3444116987 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7facd0040950 con 0x7face00f5410 2024-01-18T06:55:17.542 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:17.542+0000 7face4bea700 1 --2- 172.21.15.194:0/3444116987 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7faccc02ffb0 0x7faccc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:17.542 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:17.542+0000 7face4bea700 1 --2- 172.21.15.194:0/3444116987 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7faccc02ffb0 0x7faccc032470 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7facdc0096a0 tx=0x7facdc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:17.545 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:17.546+0000 7facd5ffb700 1 -- 172.21.15.194:0/3444116987 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7facd00079c0 con 0x7face00f5410 2024-01-18T06:55:17.752 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:17.753+0000 7face764f700 1 -- 172.21.15.194:0/3444116987 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7face00fae60 con 0x7face00f5410 2024-01-18T06:55:17.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:17.754+0000 7facd5ffb700 1 -- 172.21.15.194:0/3444116987 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7facd000f300 con 0x7face00f5410 2024-01-18T06:55:17.753 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:55:17.753 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:55:17.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:17.755+0000 7face764f700 1 -- 172.21.15.194:0/3444116987 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7faccc02ffb0 msgr2=0x7faccc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:17.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:17.755+0000 7face764f700 1 --2- 172.21.15.194:0/3444116987 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7faccc02ffb0 0x7faccc032470 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7facdc0096a0 tx=0x7facdc006b40).stop 2024-01-18T06:55:17.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:17.755+0000 7face764f700 1 -- 172.21.15.194:0/3444116987 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7face00f5410 msgr2=0x7face0109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:17.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:17.755+0000 7face764f700 1 --2- 172.21.15.194:0/3444116987 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7face00f5410 0x7face0109430 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7facd0014000 tx=0x7facd000b040).stop 2024-01-18T06:55:17.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:17.755+0000 7face764f700 1 -- 172.21.15.194:0/3444116987 shutdown_connections 2024-01-18T06:55:17.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:17.755+0000 7face764f700 1 --2- 172.21.15.194:0/3444116987 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7faccc02ffb0 0x7faccc032470 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:17.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:17.755+0000 7face764f700 1 --2- 172.21.15.194:0/3444116987 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7face00f5410 0x7face0109430 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:17.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:17.755+0000 7face764f700 1 -- 172.21.15.194:0/3444116987 >> 172.21.15.194:0/3444116987 conn(0x7face00f0b70 msgr2=0x7face00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:17.756 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:17.755+0000 7face764f700 1 -- 172.21.15.194:0/3444116987 shutdown_connections 2024-01-18T06:55:17.756 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:17.756+0000 7face764f700 1 -- 172.21.15.194:0/3444116987 wait complete. 2024-01-18T06:55:17.756 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:55:18.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:18 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:55:17.754629+0000 mon.smithi047 (mon.0) 272 : audit [DBG] from='client.? 172.21.15.194:0/3444116987' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:55:19.137 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:55:19.137 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:55:19.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:19 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:55:18.039053+0000 mgr.smithi047.fuhzzr (mgr.14162) 210 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:55:20.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:20.256+0000 7f6e08149700 1 -- 172.21.15.194:0/3637803549 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e000f5410 msgr2=0x7f6e000f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:20.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:20.256+0000 7f6e08149700 1 --2- 172.21.15.194:0/3637803549 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e000f5410 0x7f6e000f5830 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f6df0009fb0 tx=0x7f6df0009b30).stop 2024-01-18T06:55:20.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:20.256+0000 7f6e08149700 1 -- 172.21.15.194:0/3637803549 shutdown_connections 2024-01-18T06:55:20.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:20.256+0000 7f6e08149700 1 --2- 172.21.15.194:0/3637803549 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e000f5410 0x7f6e000f5830 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:20.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:20.256+0000 7f6e08149700 1 -- 172.21.15.194:0/3637803549 >> 172.21.15.194:0/3637803549 conn(0x7f6e000f0b70 msgr2=0x7f6e000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:20.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:20.256+0000 7f6e08149700 1 -- 172.21.15.194:0/3637803549 shutdown_connections 2024-01-18T06:55:20.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:20.256+0000 7f6e08149700 1 -- 172.21.15.194:0/3637803549 wait complete. 2024-01-18T06:55:20.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:20.257+0000 7f6e08149700 1 Processor -- start 2024-01-18T06:55:20.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:20.257+0000 7f6e08149700 1 -- start start 2024-01-18T06:55:20.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:20.257+0000 7f6e08149700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e000f5410 0x7f6e00105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:20.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:20.257+0000 7f6e08149700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e00105b20 con 0x7f6e000f5410 2024-01-18T06:55:20.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:20.257+0000 7f6e05ee5700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e000f5410 0x7f6e00105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:20.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:20.257+0000 7f6e05ee5700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e000f5410 0x7f6e00105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:39430/0 (socket says 172.21.15.194:39430) 2024-01-18T06:55:20.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:20.257+0000 7f6e05ee5700 1 -- 172.21.15.194:0/2508963404 learned_addr learned my addr 172.21.15.194:0/2508963404 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:55:20.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:20.258+0000 7f6e05ee5700 1 -- 172.21.15.194:0/2508963404 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df0005040 con 0x7f6e000f5410 2024-01-18T06:55:20.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:20.258+0000 7f6e05ee5700 1 --2- 172.21.15.194:0/2508963404 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e000f5410 0x7f6e00105630 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f6df0013510 tx=0x7f6df000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:20.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:20.258+0000 7f6df67fc700 1 -- 172.21.15.194:0/2508963404 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6df0009320 con 0x7f6e000f5410 2024-01-18T06:55:20.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:20.258+0000 7f6e08149700 1 -- 172.21.15.194:0/2508963404 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e00105d20 con 0x7f6e000f5410 2024-01-18T06:55:20.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:20.258+0000 7f6df67fc700 1 -- 172.21.15.194:0/2508963404 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6df0013f00 con 0x7f6e000f5410 2024-01-18T06:55:20.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:20.258+0000 7f6df67fc700 1 -- 172.21.15.194:0/2508963404 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6df0014d90 con 0x7f6e000f5410 2024-01-18T06:55:20.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:20.258+0000 7f6e08149700 1 -- 172.21.15.194:0/2508963404 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e001061b0 con 0x7f6e000f5410 2024-01-18T06:55:20.260 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:20.259+0000 7f6df67fc700 1 -- 172.21.15.194:0/2508963404 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6df00095f0 con 0x7f6e000f5410 2024-01-18T06:55:20.260 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:20.259+0000 7f6e08149700 1 -- 172.21.15.194:0/2508963404 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e00045ad0 con 0x7f6e000f5410 2024-01-18T06:55:20.260 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:20.259+0000 7f6df67fc700 1 --2- 172.21.15.194:0/2508963404 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f6dec02ff60 0x7f6dec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:20.260 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:20.259+0000 7f6df67fc700 1 -- 172.21.15.194:0/2508963404 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6df0041090 con 0x7f6e000f5410 2024-01-18T06:55:20.260 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:20.259+0000 7f6e056e4700 1 --2- 172.21.15.194:0/2508963404 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f6dec02ff60 0x7f6dec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:20.260 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:20.260+0000 7f6e056e4700 1 --2- 172.21.15.194:0/2508963404 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f6dec02ff60 0x7f6dec032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f6dfc0096a0 tx=0x7f6dfc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:20.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:20.264+0000 7f6df67fc700 1 -- 172.21.15.194:0/2508963404 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6df0014420 con 0x7f6e000f5410 2024-01-18T06:55:20.467 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:20.467+0000 7f6e08149700 1 -- 172.21.15.194:0/2508963404 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6e000fae60 con 0x7f6e000f5410 2024-01-18T06:55:20.467 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:20.468+0000 7f6df67fc700 1 -- 172.21.15.194:0/2508963404 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6df001d7f0 con 0x7f6e000f5410 2024-01-18T06:55:20.467 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:55:20.467 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:55:20.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:20.469+0000 7f6e08149700 1 -- 172.21.15.194:0/2508963404 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f6dec02ff60 msgr2=0x7f6dec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:20.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:20.469+0000 7f6e08149700 1 --2- 172.21.15.194:0/2508963404 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f6dec02ff60 0x7f6dec032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f6dfc0096a0 tx=0x7f6dfc006b40).stop 2024-01-18T06:55:20.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:20.469+0000 7f6e08149700 1 -- 172.21.15.194:0/2508963404 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e000f5410 msgr2=0x7f6e00105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:20.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:20.469+0000 7f6e08149700 1 --2- 172.21.15.194:0/2508963404 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e000f5410 0x7f6e00105630 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f6df0013510 tx=0x7f6df000b040).stop 2024-01-18T06:55:20.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:20.469+0000 7f6e08149700 1 -- 172.21.15.194:0/2508963404 shutdown_connections 2024-01-18T06:55:20.470 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:20.469+0000 7f6e08149700 1 --2- 172.21.15.194:0/2508963404 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f6dec02ff60 0x7f6dec032420 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:20.470 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:20.469+0000 7f6e08149700 1 --2- 172.21.15.194:0/2508963404 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e000f5410 0x7f6e00105630 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:20.470 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:20.469+0000 7f6e08149700 1 -- 172.21.15.194:0/2508963404 >> 172.21.15.194:0/2508963404 conn(0x7f6e000f0b70 msgr2=0x7f6e000f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:20.470 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:20.469+0000 7f6e08149700 1 -- 172.21.15.194:0/2508963404 shutdown_connections 2024-01-18T06:55:20.470 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:20.469+0000 7f6e08149700 1 -- 172.21.15.194:0/2508963404 wait complete. 2024-01-18T06:55:20.470 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:55:21.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:21 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:55:20.039442+0000 mgr.smithi047.fuhzzr (mgr.14162) 211 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:55:21.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:21 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:55:20.468796+0000 mon.smithi047 (mon.0) 273 : audit [DBG] from='client.? 172.21.15.194:0/2508963404' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:55:21.843 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:55:21.844 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:55:23.037 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:23.036+0000 7f9d00f19700 1 -- 172.21.15.194:0/2605296124 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9cfc060280 msgr2=0x7f9cfc0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:23.038 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:23.036+0000 7f9d00f19700 1 --2- 172.21.15.194:0/2605296124 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9cfc060280 0x7f9cfc0f90f0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f9cf0009fb0 tx=0x7f9cf0009b30).stop 2024-01-18T06:55:23.038 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:23.036+0000 7f9d00f19700 1 -- 172.21.15.194:0/2605296124 shutdown_connections 2024-01-18T06:55:23.038 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:23.036+0000 7f9d00f19700 1 --2- 172.21.15.194:0/2605296124 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9cfc060280 0x7f9cfc0f90f0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:23.038 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:23.036+0000 7f9d00f19700 1 -- 172.21.15.194:0/2605296124 >> 172.21.15.194:0/2605296124 conn(0x7f9cfc0f0b70 msgr2=0x7f9cfc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:23.038 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:23.036+0000 7f9d00f19700 1 -- 172.21.15.194:0/2605296124 shutdown_connections 2024-01-18T06:55:23.038 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:23.036+0000 7f9d00f19700 1 -- 172.21.15.194:0/2605296124 wait complete. 2024-01-18T06:55:23.038 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:23.037+0000 7f9d00f19700 1 Processor -- start 2024-01-18T06:55:23.038 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:23.037+0000 7f9d00f19700 1 -- start start 2024-01-18T06:55:23.038 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:23.037+0000 7f9d00f19700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9cfc060280 0x7f9cfc1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:23.038 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:23.037+0000 7f9d00f19700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc106bd0 con 0x7f9cfc060280 2024-01-18T06:55:23.038 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:23.037+0000 7f9cfa59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9cfc060280 0x7f9cfc1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:23.038 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:23.037+0000 7f9cfa59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9cfc060280 0x7f9cfc1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:39448/0 (socket says 172.21.15.194:39448) 2024-01-18T06:55:23.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:23.037+0000 7f9cfa59c700 1 -- 172.21.15.194:0/3787458175 learned_addr learned my addr 172.21.15.194:0/3787458175 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:55:23.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:23.038+0000 7f9cfa59c700 1 -- 172.21.15.194:0/3787458175 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf0005040 con 0x7f9cfc060280 2024-01-18T06:55:23.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:23.038+0000 7f9cfa59c700 1 --2- 172.21.15.194:0/3787458175 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9cfc060280 0x7f9cfc1066e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f9cf0013510 tx=0x7f9cf000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:23.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:23.038+0000 7f9ceaffd700 1 -- 172.21.15.194:0/3787458175 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cf0009320 con 0x7f9cfc060280 2024-01-18T06:55:23.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:23.038+0000 7f9d00f19700 1 -- 172.21.15.194:0/3787458175 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cfc106dd0 con 0x7f9cfc060280 2024-01-18T06:55:23.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:23.038+0000 7f9ceaffd700 1 -- 172.21.15.194:0/3787458175 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9cf0013f00 con 0x7f9cfc060280 2024-01-18T06:55:23.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:23.038+0000 7f9d00f19700 1 -- 172.21.15.194:0/3787458175 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cfc107260 con 0x7f9cfc060280 2024-01-18T06:55:23.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:23.038+0000 7f9ceaffd700 1 -- 172.21.15.194:0/3787458175 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cf0014dc0 con 0x7f9cfc060280 2024-01-18T06:55:23.040 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:23.039+0000 7f9d00f19700 1 -- 172.21.15.194:0/3787458175 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cfc045ad0 con 0x7f9cfc060280 2024-01-18T06:55:23.040 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:23.039+0000 7f9ceaffd700 1 -- 172.21.15.194:0/3787458175 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9cf00095f0 con 0x7f9cfc060280 2024-01-18T06:55:23.040 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:23.039+0000 7f9ceaffd700 1 --2- 172.21.15.194:0/3787458175 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f9ce402ff60 0x7f9ce4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:23.040 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:23.039+0000 7f9ceaffd700 1 -- 172.21.15.194:0/3787458175 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9cf0041140 con 0x7f9cfc060280 2024-01-18T06:55:23.040 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:23.039+0000 7f9cf9d9b700 1 --2- 172.21.15.194:0/3787458175 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f9ce402ff60 0x7f9ce4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:23.040 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:23.040+0000 7f9cf9d9b700 1 --2- 172.21.15.194:0/3787458175 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f9ce402ff60 0x7f9ce4032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f9cec0096a0 tx=0x7f9cec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:23.043 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:23.044+0000 7f9ceaffd700 1 -- 172.21.15.194:0/3787458175 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9cf0014420 con 0x7f9cfc060280 2024-01-18T06:55:23.252 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:23.253+0000 7f9d00f19700 1 -- 172.21.15.194:0/3787458175 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9cfc0fbf50 con 0x7f9cfc060280 2024-01-18T06:55:23.253 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:23.254+0000 7f9ceaffd700 1 -- 172.21.15.194:0/3787458175 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9cf001c790 con 0x7f9cfc060280 2024-01-18T06:55:23.253 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:55:23.253 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:55:23.255 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:23.255+0000 7f9d00f19700 1 -- 172.21.15.194:0/3787458175 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f9ce402ff60 msgr2=0x7f9ce4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:23.255 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:23.255+0000 7f9d00f19700 1 --2- 172.21.15.194:0/3787458175 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f9ce402ff60 0x7f9ce4032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f9cec0096a0 tx=0x7f9cec006b40).stop 2024-01-18T06:55:23.255 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:23.255+0000 7f9d00f19700 1 -- 172.21.15.194:0/3787458175 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9cfc060280 msgr2=0x7f9cfc1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:23.255 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:23.255+0000 7f9d00f19700 1 --2- 172.21.15.194:0/3787458175 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9cfc060280 0x7f9cfc1066e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f9cf0013510 tx=0x7f9cf000b040).stop 2024-01-18T06:55:23.255 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:23.255+0000 7f9d00f19700 1 -- 172.21.15.194:0/3787458175 shutdown_connections 2024-01-18T06:55:23.255 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:23.255+0000 7f9d00f19700 1 --2- 172.21.15.194:0/3787458175 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f9ce402ff60 0x7f9ce4032420 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:23.255 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:23.255+0000 7f9d00f19700 1 --2- 172.21.15.194:0/3787458175 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9cfc060280 0x7f9cfc1066e0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:23.255 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:23.255+0000 7f9d00f19700 1 -- 172.21.15.194:0/3787458175 >> 172.21.15.194:0/3787458175 conn(0x7f9cfc0f0b70 msgr2=0x7f9cfc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:23.255 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:23.255+0000 7f9d00f19700 1 -- 172.21.15.194:0/3787458175 shutdown_connections 2024-01-18T06:55:23.255 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:23.255+0000 7f9d00f19700 1 -- 172.21.15.194:0/3787458175 wait complete. 2024-01-18T06:55:23.256 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:55:23.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:23 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:55:22.039827+0000 mgr.smithi047.fuhzzr (mgr.14162) 212 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:55:24.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:24 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:55:23.254780+0000 mon.smithi047 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.194:0/3787458175' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:55:24.659 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:55:24.659 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:55:25.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:25 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:55:24.040132+0000 mgr.smithi047.fuhzzr (mgr.14162) 213 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:55:25.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:25.829+0000 7f4342e7f700 1 -- 172.21.15.194:0/315661794 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f433c0f51a0 msgr2=0x7f433c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:25.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:25.829+0000 7f4342e7f700 1 --2- 172.21.15.194:0/315661794 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f433c0f51a0 0x7f433c0f55c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f4338009fb0 tx=0x7f4338009b30).stop 2024-01-18T06:55:25.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:25.829+0000 7f4342e7f700 1 -- 172.21.15.194:0/315661794 shutdown_connections 2024-01-18T06:55:25.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:25.829+0000 7f4342e7f700 1 --2- 172.21.15.194:0/315661794 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f433c0f51a0 0x7f433c0f55c0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:25.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:25.829+0000 7f4342e7f700 1 -- 172.21.15.194:0/315661794 >> 172.21.15.194:0/315661794 conn(0x7f433c0f09a0 msgr2=0x7f433c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:25.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:25.829+0000 7f4342e7f700 1 -- 172.21.15.194:0/315661794 shutdown_connections 2024-01-18T06:55:25.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:25.829+0000 7f4342e7f700 1 -- 172.21.15.194:0/315661794 wait complete. 2024-01-18T06:55:25.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:25.829+0000 7f4342e7f700 1 Processor -- start 2024-01-18T06:55:25.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:25.829+0000 7f4342e7f700 1 -- start start 2024-01-18T06:55:25.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:25.829+0000 7f4342e7f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f433c0f51a0 0x7f433c10b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:25.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:25.829+0000 7f4342e7f700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f433c106fc0 con 0x7f433c0f51a0 2024-01-18T06:55:25.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:25.830+0000 7f4341e7d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f433c0f51a0 0x7f433c10b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:25.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:25.830+0000 7f4341e7d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f433c0f51a0 0x7f433c10b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:39460/0 (socket says 172.21.15.194:39460) 2024-01-18T06:55:25.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:25.830+0000 7f4341e7d700 1 -- 172.21.15.194:0/1631194886 learned_addr learned my addr 172.21.15.194:0/1631194886 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:55:25.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:25.830+0000 7f4341e7d700 1 -- 172.21.15.194:0/1631194886 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4338005040 con 0x7f433c0f51a0 2024-01-18T06:55:25.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:25.830+0000 7f4341e7d700 1 --2- 172.21.15.194:0/1631194886 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f433c0f51a0 0x7f433c10b5d0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f4338013510 tx=0x7f433800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:25.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:25.830+0000 7f432e7fc700 1 -- 172.21.15.194:0/1631194886 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4338009320 con 0x7f433c0f51a0 2024-01-18T06:55:25.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:25.830+0000 7f432e7fc700 1 -- 172.21.15.194:0/1631194886 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4338013f00 con 0x7f433c0f51a0 2024-01-18T06:55:25.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:25.830+0000 7f4342e7f700 1 -- 172.21.15.194:0/1631194886 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f433c060220 con 0x7f433c0f51a0 2024-01-18T06:55:25.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:25.830+0000 7f432e7fc700 1 -- 172.21.15.194:0/1631194886 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4338014dc0 con 0x7f433c0f51a0 2024-01-18T06:55:25.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:25.830+0000 7f4342e7f700 1 -- 172.21.15.194:0/1631194886 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f433c10bac0 con 0x7f433c0f51a0 2024-01-18T06:55:25.833 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:25.831+0000 7f432e7fc700 1 -- 172.21.15.194:0/1631194886 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4338014420 con 0x7f433c0f51a0 2024-01-18T06:55:25.833 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:25.831+0000 7f4342e7f700 1 -- 172.21.15.194:0/1631194886 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f433c045ad0 con 0x7f433c0f51a0 2024-01-18T06:55:25.833 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:25.831+0000 7f432e7fc700 1 --2- 172.21.15.194:0/1631194886 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f432802ff60 0x7f4328032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:25.833 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:25.831+0000 7f432e7fc700 1 -- 172.21.15.194:0/1631194886 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f43380410f0 con 0x7f433c0f51a0 2024-01-18T06:55:25.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:25.832+0000 7f434167c700 1 --2- 172.21.15.194:0/1631194886 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f432802ff60 0x7f4328032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:25.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:25.832+0000 7f434167c700 1 --2- 172.21.15.194:0/1631194886 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f432802ff60 0x7f4328032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f43300096a0 tx=0x7f4330006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:25.836 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:25.836+0000 7f432e7fc700 1 -- 172.21.15.194:0/1631194886 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f43380095f0 con 0x7f433c0f51a0 2024-01-18T06:55:26.041 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:26.041+0000 7f4342e7f700 1 -- 172.21.15.194:0/1631194886 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f433c10c280 con 0x7f433c0f51a0 2024-01-18T06:55:26.042 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:26.042+0000 7f432e7fc700 1 -- 172.21.15.194:0/1631194886 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f433801cb10 con 0x7f433c0f51a0 2024-01-18T06:55:26.042 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:55:26.043 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:55:26.044 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:26.043+0000 7f4342e7f700 1 -- 172.21.15.194:0/1631194886 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f432802ff60 msgr2=0x7f4328032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:26.044 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:26.043+0000 7f4342e7f700 1 --2- 172.21.15.194:0/1631194886 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f432802ff60 0x7f4328032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f43300096a0 tx=0x7f4330006b40).stop 2024-01-18T06:55:26.044 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:26.043+0000 7f4342e7f700 1 -- 172.21.15.194:0/1631194886 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f433c0f51a0 msgr2=0x7f433c10b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:26.044 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:26.043+0000 7f4342e7f700 1 --2- 172.21.15.194:0/1631194886 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f433c0f51a0 0x7f433c10b5d0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f4338013510 tx=0x7f433800b040).stop 2024-01-18T06:55:26.044 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:26.043+0000 7f4342e7f700 1 -- 172.21.15.194:0/1631194886 shutdown_connections 2024-01-18T06:55:26.045 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:26.043+0000 7f4342e7f700 1 --2- 172.21.15.194:0/1631194886 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f432802ff60 0x7f4328032420 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:26.045 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:26.043+0000 7f4342e7f700 1 --2- 172.21.15.194:0/1631194886 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f433c0f51a0 0x7f433c10b5d0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:26.045 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:26.043+0000 7f4342e7f700 1 -- 172.21.15.194:0/1631194886 >> 172.21.15.194:0/1631194886 conn(0x7f433c0f09a0 msgr2=0x7f433c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:26.045 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:26.043+0000 7f4342e7f700 1 -- 172.21.15.194:0/1631194886 shutdown_connections 2024-01-18T06:55:26.045 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:26.043+0000 7f4342e7f700 1 -- 172.21.15.194:0/1631194886 wait complete. 2024-01-18T06:55:26.046 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:55:26.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:26 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:55:26.043709+0000 mon.smithi047 (mon.0) 275 : audit [DBG] from='client.? 172.21.15.194:0/1631194886' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:55:27.451 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:55:27.452 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:55:27.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:27 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:55:26.040426+0000 mgr.smithi047.fuhzzr (mgr.14162) 214 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:55:28.588 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:28.586+0000 7f16f1a21700 1 -- 172.21.15.194:0/2184682582 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16ec0f5410 msgr2=0x7f16ec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:28.589 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:28.586+0000 7f16f1a21700 1 --2- 172.21.15.194:0/2184682582 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16ec0f5410 0x7f16ec0f5830 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f16e0009fb0 tx=0x7f16e0009b30).stop 2024-01-18T06:55:28.589 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:28.586+0000 7f16f1a21700 1 -- 172.21.15.194:0/2184682582 shutdown_connections 2024-01-18T06:55:28.589 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:28.586+0000 7f16f1a21700 1 --2- 172.21.15.194:0/2184682582 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16ec0f5410 0x7f16ec0f5830 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:28.589 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:28.586+0000 7f16f1a21700 1 -- 172.21.15.194:0/2184682582 >> 172.21.15.194:0/2184682582 conn(0x7f16ec0f0b70 msgr2=0x7f16ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:28.589 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:28.586+0000 7f16f1a21700 1 -- 172.21.15.194:0/2184682582 shutdown_connections 2024-01-18T06:55:28.589 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:28.586+0000 7f16f1a21700 1 -- 172.21.15.194:0/2184682582 wait complete. 2024-01-18T06:55:28.589 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:28.587+0000 7f16f1a21700 1 Processor -- start 2024-01-18T06:55:28.589 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:28.587+0000 7f16f1a21700 1 -- start start 2024-01-18T06:55:28.589 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:28.587+0000 7f16f1a21700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16ec0f5410 0x7f16ec105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:28.589 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:28.587+0000 7f16f1a21700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ec105af0 con 0x7f16ec0f5410 2024-01-18T06:55:28.589 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:28.587+0000 7f16eaffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16ec0f5410 0x7f16ec105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:28.589 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:28.587+0000 7f16eaffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16ec0f5410 0x7f16ec105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:55684/0 (socket says 172.21.15.194:55684) 2024-01-18T06:55:28.589 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:28.587+0000 7f16eaffd700 1 -- 172.21.15.194:0/1651609016 learned_addr learned my addr 172.21.15.194:0/1651609016 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:55:28.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:28.588+0000 7f16eaffd700 1 -- 172.21.15.194:0/1651609016 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16e0005040 con 0x7f16ec0f5410 2024-01-18T06:55:28.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:28.588+0000 7f16eaffd700 1 --2- 172.21.15.194:0/1651609016 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16ec0f5410 0x7f16ec105600 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f16e0014000 tx=0x7f16e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:28.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:28.588+0000 7f16d37fe700 1 -- 172.21.15.194:0/1651609016 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16e00095f0 con 0x7f16ec0f5410 2024-01-18T06:55:28.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:28.588+0000 7f16f1a21700 1 -- 172.21.15.194:0/1651609016 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16ec105cf0 con 0x7f16ec0f5410 2024-01-18T06:55:28.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:28.588+0000 7f16d37fe700 1 -- 172.21.15.194:0/1651609016 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f16e0007e70 con 0x7f16ec0f5410 2024-01-18T06:55:28.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:28.588+0000 7f16d37fe700 1 -- 172.21.15.194:0/1651609016 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16e0014ea0 con 0x7f16ec0f5410 2024-01-18T06:55:28.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:28.588+0000 7f16f1a21700 1 -- 172.21.15.194:0/1651609016 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16ec106180 con 0x7f16ec0f5410 2024-01-18T06:55:28.591 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:28.589+0000 7f16d37fe700 1 -- 172.21.15.194:0/1651609016 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f16e0025070 con 0x7f16ec0f5410 2024-01-18T06:55:28.591 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:28.589+0000 7f16f1a21700 1 -- 172.21.15.194:0/1651609016 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16ec045ad0 con 0x7f16ec0f5410 2024-01-18T06:55:28.591 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:28.589+0000 7f16d37fe700 1 --2- 172.21.15.194:0/1651609016 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f16d402ff60 0x7f16d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:28.591 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:28.589+0000 7f16d37fe700 1 -- 172.21.15.194:0/1651609016 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f16e0040970 con 0x7f16ec0f5410 2024-01-18T06:55:28.591 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:28.589+0000 7f16ea7fc700 1 --2- 172.21.15.194:0/1651609016 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f16d402ff60 0x7f16d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:28.591 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:28.589+0000 7f16ea7fc700 1 --2- 172.21.15.194:0/1651609016 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f16d402ff60 0x7f16d4032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f16dc0096a0 tx=0x7f16dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:28.594 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:28.594+0000 7f16d37fe700 1 -- 172.21.15.194:0/1651609016 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f16e00079c0 con 0x7f16ec0f5410 2024-01-18T06:55:28.801 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:28.801+0000 7f16f1a21700 1 -- 172.21.15.194:0/1651609016 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f16ec0fae60 con 0x7f16ec0f5410 2024-01-18T06:55:28.802 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:28.802+0000 7f16d37fe700 1 -- 172.21.15.194:0/1651609016 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f16e000f300 con 0x7f16ec0f5410 2024-01-18T06:55:28.802 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:55:28.802 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:55:28.804 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:28.803+0000 7f16f1a21700 1 -- 172.21.15.194:0/1651609016 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f16d402ff60 msgr2=0x7f16d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:28.804 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:28.803+0000 7f16f1a21700 1 --2- 172.21.15.194:0/1651609016 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f16d402ff60 0x7f16d4032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f16dc0096a0 tx=0x7f16dc006b40).stop 2024-01-18T06:55:28.804 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:28.803+0000 7f16f1a21700 1 -- 172.21.15.194:0/1651609016 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16ec0f5410 msgr2=0x7f16ec105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:28.804 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:28.803+0000 7f16f1a21700 1 --2- 172.21.15.194:0/1651609016 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16ec0f5410 0x7f16ec105600 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f16e0014000 tx=0x7f16e000b040).stop 2024-01-18T06:55:28.804 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:28.803+0000 7f16f1a21700 1 -- 172.21.15.194:0/1651609016 shutdown_connections 2024-01-18T06:55:28.804 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:28.803+0000 7f16f1a21700 1 --2- 172.21.15.194:0/1651609016 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f16d402ff60 0x7f16d4032420 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:28.804 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:28.803+0000 7f16f1a21700 1 --2- 172.21.15.194:0/1651609016 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16ec0f5410 0x7f16ec105600 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:28.804 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:28.803+0000 7f16f1a21700 1 -- 172.21.15.194:0/1651609016 >> 172.21.15.194:0/1651609016 conn(0x7f16ec0f0b70 msgr2=0x7f16ec0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:28.804 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:28.803+0000 7f16f1a21700 1 -- 172.21.15.194:0/1651609016 shutdown_connections 2024-01-18T06:55:28.804 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:28.803+0000 7f16f1a21700 1 -- 172.21.15.194:0/1651609016 wait complete. 2024-01-18T06:55:28.805 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:55:29.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:28 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:55:28.040656+0000 mgr.smithi047.fuhzzr (mgr.14162) 215 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:55:29.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:28 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:55:28.803670+0000 mon.smithi047 (mon.0) 276 : audit [DBG] from='client.? 172.21.15.194:0/1651609016' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:55:30.188 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:55:30.189 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:55:31.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:31.366+0000 7f9a95d17700 1 -- 172.21.15.194:0/3637687067 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9a900f6510 msgr2=0x7f9a900f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:31.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:31.366+0000 7f9a95d17700 1 --2- 172.21.15.194:0/3637687067 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9a900f6510 0x7f9a900f6930 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f9a84009fb0 tx=0x7f9a84009b30).stop 2024-01-18T06:55:31.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:31.367+0000 7f9a95d17700 1 -- 172.21.15.194:0/3637687067 shutdown_connections 2024-01-18T06:55:31.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:31.367+0000 7f9a95d17700 1 --2- 172.21.15.194:0/3637687067 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9a900f6510 0x7f9a900f6930 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:31.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:31.367+0000 7f9a95d17700 1 -- 172.21.15.194:0/3637687067 >> 172.21.15.194:0/3637687067 conn(0x7f9a900f1c70 msgr2=0x7f9a900f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:31.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:31.367+0000 7f9a95d17700 1 -- 172.21.15.194:0/3637687067 shutdown_connections 2024-01-18T06:55:31.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:31.367+0000 7f9a95d17700 1 -- 172.21.15.194:0/3637687067 wait complete. 2024-01-18T06:55:31.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:31.367+0000 7f9a95d17700 1 Processor -- start 2024-01-18T06:55:31.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:31.367+0000 7f9a95d17700 1 -- start start 2024-01-18T06:55:31.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:31.367+0000 7f9a95d17700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9a900f6510 0x7f9a901066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:31.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:31.367+0000 7f9a95d17700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a90106bc0 con 0x7f9a900f6510 2024-01-18T06:55:31.370 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:31.368+0000 7f9a8f7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9a900f6510 0x7f9a901066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:31.370 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:31.368+0000 7f9a8f7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9a900f6510 0x7f9a901066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:55700/0 (socket says 172.21.15.194:55700) 2024-01-18T06:55:31.370 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:31.368+0000 7f9a8f7fe700 1 -- 172.21.15.194:0/511050771 learned_addr learned my addr 172.21.15.194:0/511050771 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:55:31.370 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:31.368+0000 7f9a8f7fe700 1 -- 172.21.15.194:0/511050771 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a84005040 con 0x7f9a900f6510 2024-01-18T06:55:31.370 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:31.368+0000 7f9a8f7fe700 1 --2- 172.21.15.194:0/511050771 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9a900f6510 0x7f9a901066d0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f9a84013510 tx=0x7f9a8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:31.370 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:31.368+0000 7f9a77fff700 1 -- 172.21.15.194:0/511050771 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a84009320 con 0x7f9a900f6510 2024-01-18T06:55:31.370 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:31.368+0000 7f9a95d17700 1 -- 172.21.15.194:0/511050771 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a90106dc0 con 0x7f9a900f6510 2024-01-18T06:55:31.370 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:31.368+0000 7f9a77fff700 1 -- 172.21.15.194:0/511050771 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9a84013f00 con 0x7f9a900f6510 2024-01-18T06:55:31.370 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:31.368+0000 7f9a77fff700 1 -- 172.21.15.194:0/511050771 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a84014d90 con 0x7f9a900f6510 2024-01-18T06:55:31.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:31.368+0000 7f9a95d17700 1 -- 172.21.15.194:0/511050771 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a90107250 con 0x7f9a900f6510 2024-01-18T06:55:31.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:31.369+0000 7f9a77fff700 1 -- 172.21.15.194:0/511050771 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9a84014420 con 0x7f9a900f6510 2024-01-18T06:55:31.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:31.369+0000 7f9a95d17700 1 -- 172.21.15.194:0/511050771 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a90045ad0 con 0x7f9a900f6510 2024-01-18T06:55:31.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:31.369+0000 7f9a77fff700 1 --2- 172.21.15.194:0/511050771 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f9a7802ff60 0x7f9a78032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:31.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:31.369+0000 7f9a77fff700 1 -- 172.21.15.194:0/511050771 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9a840410c0 con 0x7f9a900f6510 2024-01-18T06:55:31.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:31.370+0000 7f9a8effd700 1 --2- 172.21.15.194:0/511050771 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f9a7802ff60 0x7f9a78032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:31.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:31.370+0000 7f9a8effd700 1 --2- 172.21.15.194:0/511050771 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f9a7802ff60 0x7f9a78032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f9a800096a0 tx=0x7f9a80006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:31.374 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:31.375+0000 7f9a77fff700 1 -- 172.21.15.194:0/511050771 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9a840095f0 con 0x7f9a900f6510 2024-01-18T06:55:31.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:31 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:55:30.040983+0000 mgr.smithi047.fuhzzr (mgr.14162) 216 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:55:31.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:31 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:55:30.118922+0000 mon.smithi047 (mon.0) 277 : audit [INF] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.fuhzzr/mirror_snapshot_schedule"}]: dispatch 2024-01-18T06:55:31.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:31 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:55:30.119146+0000 mon.smithi047 (mon.0) 278 : audit [INF] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.fuhzzr/trash_purge_schedule"}]: dispatch 2024-01-18T06:55:31.581 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:31.580+0000 7f9a95d17700 1 -- 172.21.15.194:0/511050771 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9a90108100 con 0x7f9a900f6510 2024-01-18T06:55:31.581 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:31.581+0000 7f9a77fff700 1 -- 172.21.15.194:0/511050771 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9a8401db70 con 0x7f9a900f6510 2024-01-18T06:55:31.581 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:55:31.582 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:55:31.583 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:31.583+0000 7f9a95d17700 1 -- 172.21.15.194:0/511050771 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f9a7802ff60 msgr2=0x7f9a78032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:31.583 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:31.583+0000 7f9a95d17700 1 --2- 172.21.15.194:0/511050771 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f9a7802ff60 0x7f9a78032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f9a800096a0 tx=0x7f9a80006b40).stop 2024-01-18T06:55:31.584 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:31.583+0000 7f9a95d17700 1 -- 172.21.15.194:0/511050771 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9a900f6510 msgr2=0x7f9a901066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:31.584 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:31.583+0000 7f9a95d17700 1 --2- 172.21.15.194:0/511050771 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9a900f6510 0x7f9a901066d0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f9a84013510 tx=0x7f9a8400b040).stop 2024-01-18T06:55:31.584 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:31.583+0000 7f9a95d17700 1 -- 172.21.15.194:0/511050771 shutdown_connections 2024-01-18T06:55:31.584 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:31.583+0000 7f9a95d17700 1 --2- 172.21.15.194:0/511050771 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f9a7802ff60 0x7f9a78032420 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:31.584 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:31.583+0000 7f9a95d17700 1 --2- 172.21.15.194:0/511050771 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9a900f6510 0x7f9a901066d0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:31.584 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:31.583+0000 7f9a95d17700 1 -- 172.21.15.194:0/511050771 >> 172.21.15.194:0/511050771 conn(0x7f9a900f1c70 msgr2=0x7f9a900f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:31.584 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:31.583+0000 7f9a95d17700 1 -- 172.21.15.194:0/511050771 shutdown_connections 2024-01-18T06:55:31.584 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:31.583+0000 7f9a95d17700 1 -- 172.21.15.194:0/511050771 wait complete. 2024-01-18T06:55:31.585 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:55:32.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:32 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:55:31.582936+0000 mon.smithi047 (mon.0) 279 : audit [DBG] from='client.? 172.21.15.194:0/511050771' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:55:32.960 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:55:32.960 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:55:33.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:33 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:55:32.041303+0000 mgr.smithi047.fuhzzr (mgr.14162) 217 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:55:34.162 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:34.162+0000 7f1b417fc700 1 -- 172.21.15.194:0/493828820 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b3c0f5410 msgr2=0x7f1b3c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:34.162 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:34.162+0000 7f1b417fc700 1 --2- 172.21.15.194:0/493828820 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b3c0f5410 0x7f1b3c0f5830 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f1b30009fb0 tx=0x7f1b30009b30).stop 2024-01-18T06:55:34.162 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:34.162+0000 7f1b417fc700 1 -- 172.21.15.194:0/493828820 shutdown_connections 2024-01-18T06:55:34.162 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:34.162+0000 7f1b417fc700 1 --2- 172.21.15.194:0/493828820 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b3c0f5410 0x7f1b3c0f5830 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:34.163 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:34.162+0000 7f1b417fc700 1 -- 172.21.15.194:0/493828820 >> 172.21.15.194:0/493828820 conn(0x7f1b3c0f0b70 msgr2=0x7f1b3c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:34.163 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:34.162+0000 7f1b417fc700 1 -- 172.21.15.194:0/493828820 shutdown_connections 2024-01-18T06:55:34.163 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:34.162+0000 7f1b417fc700 1 -- 172.21.15.194:0/493828820 wait complete. 2024-01-18T06:55:34.165 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:34.163+0000 7f1b417fc700 1 Processor -- start 2024-01-18T06:55:34.165 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:34.163+0000 7f1b417fc700 1 -- start start 2024-01-18T06:55:34.165 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:34.163+0000 7f1b417fc700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b3c0f5410 0x7f1b3c1071b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:34.165 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:34.163+0000 7f1b417fc700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3c060280 con 0x7f1b3c0f5410 2024-01-18T06:55:34.165 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:34.164+0000 7f1b3affd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b3c0f5410 0x7f1b3c1071b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:34.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:34.164+0000 7f1b3affd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b3c0f5410 0x7f1b3c1071b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:55718/0 (socket says 172.21.15.194:55718) 2024-01-18T06:55:34.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:34.164+0000 7f1b3affd700 1 -- 172.21.15.194:0/287939819 learned_addr learned my addr 172.21.15.194:0/287939819 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:55:34.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:34.164+0000 7f1b3affd700 1 -- 172.21.15.194:0/287939819 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b30005040 con 0x7f1b3c0f5410 2024-01-18T06:55:34.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:34.164+0000 7f1b3affd700 1 --2- 172.21.15.194:0/287939819 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b3c0f5410 0x7f1b3c1071b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f1b30013510 tx=0x7f1b3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:34.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:34.164+0000 7f1b237fe700 1 -- 172.21.15.194:0/287939819 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b30009320 con 0x7f1b3c0f5410 2024-01-18T06:55:34.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:34.164+0000 7f1b237fe700 1 -- 172.21.15.194:0/287939819 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1b30013f00 con 0x7f1b3c0f5410 2024-01-18T06:55:34.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:34.164+0000 7f1b417fc700 1 -- 172.21.15.194:0/287939819 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b3c060480 con 0x7f1b3c0f5410 2024-01-18T06:55:34.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:34.164+0000 7f1b417fc700 1 -- 172.21.15.194:0/287939819 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b3c1078e0 con 0x7f1b3c0f5410 2024-01-18T06:55:34.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:34.164+0000 7f1b237fe700 1 -- 172.21.15.194:0/287939819 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b30014d90 con 0x7f1b3c0f5410 2024-01-18T06:55:34.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:34.165+0000 7f1b237fe700 1 -- 172.21.15.194:0/287939819 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1b300263a0 con 0x7f1b3c0f5410 2024-01-18T06:55:34.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:34.165+0000 7f1b237fe700 1 --2- 172.21.15.194:0/287939819 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f1b2402ff60 0x7f1b24032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:34.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:34.165+0000 7f1b417fc700 1 -- 172.21.15.194:0/287939819 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b1c004fa0 con 0x7f1b3c0f5410 2024-01-18T06:55:34.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:34.165+0000 7f1b237fe700 1 -- 172.21.15.194:0/287939819 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1b30041080 con 0x7f1b3c0f5410 2024-01-18T06:55:34.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:34.166+0000 7f1b3a7fc700 1 --2- 172.21.15.194:0/287939819 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f1b2402ff60 0x7f1b24032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:34.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:34.166+0000 7f1b3a7fc700 1 --2- 172.21.15.194:0/287939819 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f1b2402ff60 0x7f1b24032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f1b2c0096a0 tx=0x7f1b2c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:34.170 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:34.170+0000 7f1b237fe700 1 -- 172.21.15.194:0/287939819 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1b30012510 con 0x7f1b3c0f5410 2024-01-18T06:55:34.374 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:34.375+0000 7f1b417fc700 1 -- 172.21.15.194:0/287939819 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1b1c005670 con 0x7f1b3c0f5410 2024-01-18T06:55:34.375 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:34.375+0000 7f1b237fe700 1 -- 172.21.15.194:0/287939819 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1b30012330 con 0x7f1b3c0f5410 2024-01-18T06:55:34.375 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:55:34.375 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:55:34.376 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:34.377+0000 7f1b417fc700 1 -- 172.21.15.194:0/287939819 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f1b2402ff60 msgr2=0x7f1b24032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:34.376 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:34.377+0000 7f1b417fc700 1 --2- 172.21.15.194:0/287939819 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f1b2402ff60 0x7f1b24032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f1b2c0096a0 tx=0x7f1b2c006b40).stop 2024-01-18T06:55:34.377 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:34.377+0000 7f1b417fc700 1 -- 172.21.15.194:0/287939819 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b3c0f5410 msgr2=0x7f1b3c1071b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:34.377 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:34.377+0000 7f1b417fc700 1 --2- 172.21.15.194:0/287939819 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b3c0f5410 0x7f1b3c1071b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f1b30013510 tx=0x7f1b3000b040).stop 2024-01-18T06:55:34.377 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:34.377+0000 7f1b417fc700 1 -- 172.21.15.194:0/287939819 shutdown_connections 2024-01-18T06:55:34.377 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:34.377+0000 7f1b417fc700 1 --2- 172.21.15.194:0/287939819 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f1b2402ff60 0x7f1b24032420 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:34.377 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:34.377+0000 7f1b417fc700 1 --2- 172.21.15.194:0/287939819 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b3c0f5410 0x7f1b3c1071b0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:34.377 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:34.377+0000 7f1b417fc700 1 -- 172.21.15.194:0/287939819 >> 172.21.15.194:0/287939819 conn(0x7f1b3c0f0b70 msgr2=0x7f1b3c0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:34.377 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:34.377+0000 7f1b417fc700 1 -- 172.21.15.194:0/287939819 shutdown_connections 2024-01-18T06:55:34.377 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:34.377+0000 7f1b417fc700 1 -- 172.21.15.194:0/287939819 wait complete. 2024-01-18T06:55:34.377 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:55:35.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:35 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:55:34.041736+0000 mgr.smithi047.fuhzzr (mgr.14162) 218 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:55:35.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:35 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:55:34.376527+0000 mon.smithi047 (mon.0) 280 : audit [DBG] from='client.? 172.21.15.194:0/287939819' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:55:35.760 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:55:35.761 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:55:36.948 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:36.947+0000 7f71adf2e700 1 -- 172.21.15.194:0/2516808677 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f71a80f5410 msgr2=0x7f71a80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:36.948 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:36.947+0000 7f71adf2e700 1 --2- 172.21.15.194:0/2516808677 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f71a80f5410 0x7f71a80f5830 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f719c009fb0 tx=0x7f719c009b30).stop 2024-01-18T06:55:36.948 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:36.947+0000 7f71adf2e700 1 -- 172.21.15.194:0/2516808677 shutdown_connections 2024-01-18T06:55:36.949 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:36.947+0000 7f71adf2e700 1 --2- 172.21.15.194:0/2516808677 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f71a80f5410 0x7f71a80f5830 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:36.949 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:36.947+0000 7f71adf2e700 1 -- 172.21.15.194:0/2516808677 >> 172.21.15.194:0/2516808677 conn(0x7f71a80f0b70 msgr2=0x7f71a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:36.949 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:36.947+0000 7f71adf2e700 1 -- 172.21.15.194:0/2516808677 shutdown_connections 2024-01-18T06:55:36.949 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:36.947+0000 7f71adf2e700 1 -- 172.21.15.194:0/2516808677 wait complete. 2024-01-18T06:55:36.949 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:36.947+0000 7f71adf2e700 1 Processor -- start 2024-01-18T06:55:36.949 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:36.948+0000 7f71adf2e700 1 -- start start 2024-01-18T06:55:36.949 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:36.948+0000 7f71adf2e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f71a80f5410 0x7f71a8109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:36.949 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:36.948+0000 7f71adf2e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71a8060220 con 0x7f71a80f5410 2024-01-18T06:55:36.949 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:36.948+0000 7f71a77fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f71a80f5410 0x7f71a8109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:36.949 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:36.948+0000 7f71a77fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f71a80f5410 0x7f71a8109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:55744/0 (socket says 172.21.15.194:55744) 2024-01-18T06:55:36.949 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:36.948+0000 7f71a77fe700 1 -- 172.21.15.194:0/473684324 learned_addr learned my addr 172.21.15.194:0/473684324 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:55:36.949 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:36.948+0000 7f71a77fe700 1 -- 172.21.15.194:0/473684324 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f719c005040 con 0x7f71a80f5410 2024-01-18T06:55:36.950 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:36.948+0000 7f71a77fe700 1 --2- 172.21.15.194:0/473684324 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f71a80f5410 0x7f71a8109450 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f719c013510 tx=0x7f719c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:36.950 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:36.949+0000 7f718ffff700 1 -- 172.21.15.194:0/473684324 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f719c009320 con 0x7f71a80f5410 2024-01-18T06:55:36.950 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:36.949+0000 7f71adf2e700 1 -- 172.21.15.194:0/473684324 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71a8060420 con 0x7f71a80f5410 2024-01-18T06:55:36.950 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:36.949+0000 7f718ffff700 1 -- 172.21.15.194:0/473684324 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f719c013f00 con 0x7f71a80f5410 2024-01-18T06:55:36.950 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:36.949+0000 7f718ffff700 1 -- 172.21.15.194:0/473684324 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f719c014d90 con 0x7f71a80f5410 2024-01-18T06:55:36.950 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:36.949+0000 7f71adf2e700 1 -- 172.21.15.194:0/473684324 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71a8109b00 con 0x7f71a80f5410 2024-01-18T06:55:36.951 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:36.949+0000 7f718ffff700 1 -- 172.21.15.194:0/473684324 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f719c0095f0 con 0x7f71a80f5410 2024-01-18T06:55:36.951 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:36.950+0000 7f71adf2e700 1 -- 172.21.15.194:0/473684324 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71a8059100 con 0x7f71a80f5410 2024-01-18T06:55:36.951 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:36.950+0000 7f718ffff700 1 --2- 172.21.15.194:0/473684324 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f719002ff60 0x7f7190032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:36.951 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:36.950+0000 7f718ffff700 1 -- 172.21.15.194:0/473684324 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f719c040fe0 con 0x7f71a80f5410 2024-01-18T06:55:36.951 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:36.950+0000 7f71a6ffd700 1 --2- 172.21.15.194:0/473684324 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f719002ff60 0x7f7190032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:36.951 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:36.950+0000 7f71a6ffd700 1 --2- 172.21.15.194:0/473684324 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f719002ff60 0x7f7190032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f71980096a0 tx=0x7f7198006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:36.954 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:36.955+0000 7f718ffff700 1 -- 172.21.15.194:0/473684324 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f719c01d7f0 con 0x7f71a80f5410 2024-01-18T06:55:37.157 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:37.158+0000 7f71adf2e700 1 -- 172.21.15.194:0/473684324 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f71a8045ad0 con 0x7f71a80f5410 2024-01-18T06:55:37.158 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:37.158+0000 7f718ffff700 1 -- 172.21.15.194:0/473684324 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f719c010230 con 0x7f71a80f5410 2024-01-18T06:55:37.158 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:55:37.158 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:55:37.160 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:37.160+0000 7f71adf2e700 1 -- 172.21.15.194:0/473684324 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f719002ff60 msgr2=0x7f7190032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:37.160 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:37.160+0000 7f71adf2e700 1 --2- 172.21.15.194:0/473684324 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f719002ff60 0x7f7190032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f71980096a0 tx=0x7f7198006b40).stop 2024-01-18T06:55:37.160 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:37.160+0000 7f71adf2e700 1 -- 172.21.15.194:0/473684324 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f71a80f5410 msgr2=0x7f71a8109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:37.160 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:37.160+0000 7f71adf2e700 1 --2- 172.21.15.194:0/473684324 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f71a80f5410 0x7f71a8109450 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f719c013510 tx=0x7f719c00b040).stop 2024-01-18T06:55:37.160 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:37.160+0000 7f71adf2e700 1 -- 172.21.15.194:0/473684324 shutdown_connections 2024-01-18T06:55:37.160 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:37.160+0000 7f71adf2e700 1 --2- 172.21.15.194:0/473684324 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f719002ff60 0x7f7190032420 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:37.160 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:37.160+0000 7f71adf2e700 1 --2- 172.21.15.194:0/473684324 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f71a80f5410 0x7f71a8109450 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:37.160 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:37.160+0000 7f71adf2e700 1 -- 172.21.15.194:0/473684324 >> 172.21.15.194:0/473684324 conn(0x7f71a80f0b70 msgr2=0x7f71a80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:37.160 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:37.160+0000 7f71adf2e700 1 -- 172.21.15.194:0/473684324 shutdown_connections 2024-01-18T06:55:37.160 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:37.160+0000 7f71adf2e700 1 -- 172.21.15.194:0/473684324 wait complete. 2024-01-18T06:55:37.161 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:55:37.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:55:36.042079+0000 mgr.smithi047.fuhzzr (mgr.14162) 219 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:55:38.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:38 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:55:37.159457+0000 mon.smithi047 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.194:0/473684324' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:55:38.545 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:55:38.546 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:55:39.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:39 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:55:38.042364+0000 mgr.smithi047.fuhzzr (mgr.14162) 220 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:55:39.751 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:39.751+0000 7fdb9df04700 1 -- 172.21.15.194:0/967313456 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdb980601c0 msgr2=0x7fdb980f91b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:39.751 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:39.751+0000 7fdb9df04700 1 --2- 172.21.15.194:0/967313456 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdb980601c0 0x7fdb980f91b0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7fdb88009fb0 tx=0x7fdb88009b30).stop 2024-01-18T06:55:39.751 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:39.751+0000 7fdb9df04700 1 -- 172.21.15.194:0/967313456 shutdown_connections 2024-01-18T06:55:39.751 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:39.751+0000 7fdb9df04700 1 --2- 172.21.15.194:0/967313456 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdb980601c0 0x7fdb980f91b0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:39.751 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:39.751+0000 7fdb9df04700 1 -- 172.21.15.194:0/967313456 >> 172.21.15.194:0/967313456 conn(0x7fdb980f0b70 msgr2=0x7fdb980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:39.752 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:39.751+0000 7fdb9df04700 1 -- 172.21.15.194:0/967313456 shutdown_connections 2024-01-18T06:55:39.752 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:39.751+0000 7fdb9df04700 1 -- 172.21.15.194:0/967313456 wait complete. 2024-01-18T06:55:39.752 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:39.751+0000 7fdb9df04700 1 Processor -- start 2024-01-18T06:55:39.752 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:39.752+0000 7fdb9df04700 1 -- start start 2024-01-18T06:55:39.752 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:39.752+0000 7fdb9df04700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdb980601c0 0x7fdb981088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:39.752 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:39.752+0000 7fdb9df04700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb98108dc0 con 0x7fdb980601c0 2024-01-18T06:55:39.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:39.752+0000 7fdb977fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdb980601c0 0x7fdb981088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:39.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:39.752+0000 7fdb977fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdb980601c0 0x7fdb981088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:56802/0 (socket says 172.21.15.194:56802) 2024-01-18T06:55:39.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:39.752+0000 7fdb977fe700 1 -- 172.21.15.194:0/3733966878 learned_addr learned my addr 172.21.15.194:0/3733966878 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:55:39.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:39.752+0000 7fdb977fe700 1 -- 172.21.15.194:0/3733966878 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb88005040 con 0x7fdb980601c0 2024-01-18T06:55:39.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:39.752+0000 7fdb977fe700 1 --2- 172.21.15.194:0/3733966878 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdb980601c0 0x7fdb981088d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7fdb88013510 tx=0x7fdb8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:39.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:39.753+0000 7fdb7ffff700 1 -- 172.21.15.194:0/3733966878 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb88009320 con 0x7fdb980601c0 2024-01-18T06:55:39.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:39.753+0000 7fdb9df04700 1 -- 172.21.15.194:0/3733966878 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb98108fc0 con 0x7fdb980601c0 2024-01-18T06:55:39.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:39.753+0000 7fdb7ffff700 1 -- 172.21.15.194:0/3733966878 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdb88013f00 con 0x7fdb980601c0 2024-01-18T06:55:39.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:39.753+0000 7fdb9df04700 1 -- 172.21.15.194:0/3733966878 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb98109450 con 0x7fdb980601c0 2024-01-18T06:55:39.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:39.753+0000 7fdb7ffff700 1 -- 172.21.15.194:0/3733966878 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb88014dc0 con 0x7fdb980601c0 2024-01-18T06:55:39.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:39.754+0000 7fdb7ffff700 1 -- 172.21.15.194:0/3733966878 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdb8801d3d0 con 0x7fdb980601c0 2024-01-18T06:55:39.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:39.754+0000 7fdb7ffff700 1 --2- 172.21.15.194:0/3733966878 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fdb8002ff60 0x7fdb80032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:39.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:39.754+0000 7fdb7ffff700 1 -- 172.21.15.194:0/3733966878 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdb88041090 con 0x7fdb980601c0 2024-01-18T06:55:39.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:39.754+0000 7fdb9df04700 1 -- 172.21.15.194:0/3733966878 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb98045ad0 con 0x7fdb980601c0 2024-01-18T06:55:39.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:39.754+0000 7fdb96ffd700 1 --2- 172.21.15.194:0/3733966878 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fdb8002ff60 0x7fdb80032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:39.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:39.754+0000 7fdb96ffd700 1 --2- 172.21.15.194:0/3733966878 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fdb8002ff60 0x7fdb80032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fdb8c0096a0 tx=0x7fdb8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:39.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:39.759+0000 7fdb7ffff700 1 -- 172.21.15.194:0/3733966878 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdb88012920 con 0x7fdb980601c0 2024-01-18T06:55:39.964 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:39.964+0000 7fdb9df04700 1 -- 172.21.15.194:0/3733966878 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdb980fbf50 con 0x7fdb980601c0 2024-01-18T06:55:39.965 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:39.965+0000 7fdb7ffff700 1 -- 172.21.15.194:0/3733966878 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdb88012330 con 0x7fdb980601c0 2024-01-18T06:55:39.965 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:55:39.965 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:55:39.966 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:39.967+0000 7fdb9df04700 1 -- 172.21.15.194:0/3733966878 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fdb8002ff60 msgr2=0x7fdb80032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:39.966 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:39.967+0000 7fdb9df04700 1 --2- 172.21.15.194:0/3733966878 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fdb8002ff60 0x7fdb80032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fdb8c0096a0 tx=0x7fdb8c006b40).stop 2024-01-18T06:55:39.966 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:39.967+0000 7fdb9df04700 1 -- 172.21.15.194:0/3733966878 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdb980601c0 msgr2=0x7fdb981088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:39.967 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:39.967+0000 7fdb9df04700 1 --2- 172.21.15.194:0/3733966878 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdb980601c0 0x7fdb981088d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7fdb88013510 tx=0x7fdb8800b040).stop 2024-01-18T06:55:39.967 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:39.967+0000 7fdb9df04700 1 -- 172.21.15.194:0/3733966878 shutdown_connections 2024-01-18T06:55:39.967 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:39.967+0000 7fdb9df04700 1 --2- 172.21.15.194:0/3733966878 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fdb8002ff60 0x7fdb80032420 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:39.967 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:39.967+0000 7fdb9df04700 1 --2- 172.21.15.194:0/3733966878 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdb980601c0 0x7fdb981088d0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:39.967 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:39.967+0000 7fdb9df04700 1 -- 172.21.15.194:0/3733966878 >> 172.21.15.194:0/3733966878 conn(0x7fdb980f0b70 msgr2=0x7fdb980f1650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:39.967 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:39.967+0000 7fdb9df04700 1 -- 172.21.15.194:0/3733966878 shutdown_connections 2024-01-18T06:55:39.967 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:39.967+0000 7fdb9df04700 1 -- 172.21.15.194:0/3733966878 wait complete. 2024-01-18T06:55:39.968 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:55:40.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:40 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:55:39.966147+0000 mon.smithi047 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.194:0/3733966878' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:55:41.355 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:55:41.355 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:55:41.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:41 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:55:40.042682+0000 mgr.smithi047.fuhzzr (mgr.14162) 221 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:55:42.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:42.566+0000 7fb0e0b3c700 1 -- 172.21.15.194:0/1428991161 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0dc0f5410 msgr2=0x7fb0dc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:42.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:42.566+0000 7fb0e0b3c700 1 --2- 172.21.15.194:0/1428991161 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0dc0f5410 0x7fb0dc0f5830 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7fb0cc009fb0 tx=0x7fb0cc009b30).stop 2024-01-18T06:55:42.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:42.567+0000 7fb0e0b3c700 1 -- 172.21.15.194:0/1428991161 shutdown_connections 2024-01-18T06:55:42.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:42.567+0000 7fb0e0b3c700 1 --2- 172.21.15.194:0/1428991161 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0dc0f5410 0x7fb0dc0f5830 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:42.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:42.567+0000 7fb0e0b3c700 1 -- 172.21.15.194:0/1428991161 >> 172.21.15.194:0/1428991161 conn(0x7fb0dc0f0b70 msgr2=0x7fb0dc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:42.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:42.567+0000 7fb0e0b3c700 1 -- 172.21.15.194:0/1428991161 shutdown_connections 2024-01-18T06:55:42.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:42.567+0000 7fb0e0b3c700 1 -- 172.21.15.194:0/1428991161 wait complete. 2024-01-18T06:55:42.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:42.567+0000 7fb0e0b3c700 1 Processor -- start 2024-01-18T06:55:42.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:42.567+0000 7fb0e0b3c700 1 -- start start 2024-01-18T06:55:42.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:42.567+0000 7fb0e0b3c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0dc0f5410 0x7fb0dc105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:42.568 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:42.567+0000 7fb0e0b3c700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0dc105b20 con 0x7fb0dc0f5410 2024-01-18T06:55:42.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:42.568+0000 7fb0da59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0dc0f5410 0x7fb0dc105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:42.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:42.568+0000 7fb0da59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0dc0f5410 0x7fb0dc105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:56824/0 (socket says 172.21.15.194:56824) 2024-01-18T06:55:42.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:42.568+0000 7fb0da59c700 1 -- 172.21.15.194:0/3958312716 learned_addr learned my addr 172.21.15.194:0/3958312716 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:55:42.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:42.568+0000 7fb0da59c700 1 -- 172.21.15.194:0/3958312716 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0cc005040 con 0x7fb0dc0f5410 2024-01-18T06:55:42.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:42.568+0000 7fb0da59c700 1 --2- 172.21.15.194:0/3958312716 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0dc0f5410 0x7fb0dc105630 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7fb0cc013510 tx=0x7fb0cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:42.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:42.568+0000 7fb0d2ffd700 1 -- 172.21.15.194:0/3958312716 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0cc009320 con 0x7fb0dc0f5410 2024-01-18T06:55:42.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:42.568+0000 7fb0e0b3c700 1 -- 172.21.15.194:0/3958312716 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0dc105d20 con 0x7fb0dc0f5410 2024-01-18T06:55:42.570 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:42.568+0000 7fb0d2ffd700 1 -- 172.21.15.194:0/3958312716 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb0cc013f00 con 0x7fb0dc0f5410 2024-01-18T06:55:42.570 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:42.568+0000 7fb0d2ffd700 1 -- 172.21.15.194:0/3958312716 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0cc014d90 con 0x7fb0dc0f5410 2024-01-18T06:55:42.570 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:42.568+0000 7fb0e0b3c700 1 -- 172.21.15.194:0/3958312716 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0dc1061b0 con 0x7fb0dc0f5410 2024-01-18T06:55:42.570 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:42.569+0000 7fb0d2ffd700 1 -- 172.21.15.194:0/3958312716 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb0cc0095f0 con 0x7fb0dc0f5410 2024-01-18T06:55:42.570 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:42.569+0000 7fb0e0b3c700 1 -- 172.21.15.194:0/3958312716 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0dc045ad0 con 0x7fb0dc0f5410 2024-01-18T06:55:42.570 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:42.570+0000 7fb0d2ffd700 1 --2- 172.21.15.194:0/3958312716 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb0c802ff60 0x7fb0c8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:42.570 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:42.570+0000 7fb0d2ffd700 1 -- 172.21.15.194:0/3958312716 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb0cc040fe0 con 0x7fb0dc0f5410 2024-01-18T06:55:42.570 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:42.570+0000 7fb0d9d9b700 1 --2- 172.21.15.194:0/3958312716 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb0c802ff60 0x7fb0c8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:42.570 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:42.570+0000 7fb0d9d9b700 1 --2- 172.21.15.194:0/3958312716 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb0c802ff60 0x7fb0c8032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fb0c40096a0 tx=0x7fb0c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:42.574 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:42.574+0000 7fb0d2ffd700 1 -- 172.21.15.194:0/3958312716 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb0cc01d7f0 con 0x7fb0dc0f5410 2024-01-18T06:55:42.778 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:42.778+0000 7fb0e0b3c700 1 -- 172.21.15.194:0/3958312716 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb0dc0fae60 con 0x7fb0dc0f5410 2024-01-18T06:55:42.778 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:42.779+0000 7fb0d2ffd700 1 -- 172.21.15.194:0/3958312716 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb0cc010230 con 0x7fb0dc0f5410 2024-01-18T06:55:42.779 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:55:42.779 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:55:42.780 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:42.781+0000 7fb0e0b3c700 1 -- 172.21.15.194:0/3958312716 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb0c802ff60 msgr2=0x7fb0c8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:42.780 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:42.781+0000 7fb0e0b3c700 1 --2- 172.21.15.194:0/3958312716 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb0c802ff60 0x7fb0c8032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fb0c40096a0 tx=0x7fb0c4006b40).stop 2024-01-18T06:55:42.780 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:42.781+0000 7fb0e0b3c700 1 -- 172.21.15.194:0/3958312716 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0dc0f5410 msgr2=0x7fb0dc105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:42.781 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:42.781+0000 7fb0e0b3c700 1 --2- 172.21.15.194:0/3958312716 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0dc0f5410 0x7fb0dc105630 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7fb0cc013510 tx=0x7fb0cc00b040).stop 2024-01-18T06:55:42.781 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:42.781+0000 7fb0e0b3c700 1 -- 172.21.15.194:0/3958312716 shutdown_connections 2024-01-18T06:55:42.781 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:42.781+0000 7fb0e0b3c700 1 --2- 172.21.15.194:0/3958312716 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb0c802ff60 0x7fb0c8032420 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:42.781 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:42.781+0000 7fb0e0b3c700 1 --2- 172.21.15.194:0/3958312716 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0dc0f5410 0x7fb0dc105630 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:42.781 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:42.781+0000 7fb0e0b3c700 1 -- 172.21.15.194:0/3958312716 >> 172.21.15.194:0/3958312716 conn(0x7fb0dc0f0b70 msgr2=0x7fb0dc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:42.781 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:42.781+0000 7fb0e0b3c700 1 -- 172.21.15.194:0/3958312716 shutdown_connections 2024-01-18T06:55:42.782 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:42.781+0000 7fb0e0b3c700 1 -- 172.21.15.194:0/3958312716 wait complete. 2024-01-18T06:55:42.782 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:55:43.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:43 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:55:42.042938+0000 mgr.smithi047.fuhzzr (mgr.14162) 222 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:55:43.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:43 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:55:42.780274+0000 mon.smithi047 (mon.0) 283 : audit [DBG] from='client.? 172.21.15.194:0/3958312716' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:55:44.159 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:55:44.160 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:55:45.420 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:45.419+0000 7f2d8359e700 1 -- 172.21.15.194:0/3168645321 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d84060140 msgr2=0x7f2d840f8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:45.420 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:45.419+0000 7f2d8359e700 1 --2- 172.21.15.194:0/3168645321 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d84060140 0x7f2d840f8fc0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f2d74009fb0 tx=0x7f2d74009b30).stop 2024-01-18T06:55:45.420 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:45.419+0000 7f2d8359e700 1 -- 172.21.15.194:0/3168645321 shutdown_connections 2024-01-18T06:55:45.420 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:45.419+0000 7f2d8359e700 1 --2- 172.21.15.194:0/3168645321 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d84060140 0x7f2d840f8fc0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:45.420 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:45.419+0000 7f2d8359e700 1 -- 172.21.15.194:0/3168645321 >> 172.21.15.194:0/3168645321 conn(0x7f2d840f09a0 msgr2=0x7f2d840f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:45.420 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:45.419+0000 7f2d8359e700 1 -- 172.21.15.194:0/3168645321 shutdown_connections 2024-01-18T06:55:45.421 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:45.419+0000 7f2d8359e700 1 -- 172.21.15.194:0/3168645321 wait complete. 2024-01-18T06:55:45.421 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:45.419+0000 7f2d8359e700 1 Processor -- start 2024-01-18T06:55:45.421 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:45.419+0000 7f2d8359e700 1 -- start start 2024-01-18T06:55:45.421 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:45.419+0000 7f2d8359e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d84060140 0x7f2d84106520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:45.421 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:45.419+0000 7f2d8359e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d84106a10 con 0x7f2d84060140 2024-01-18T06:55:45.421 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:45.420+0000 7f2d8259c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d84060140 0x7f2d84106520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:45.421 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:45.420+0000 7f2d8259c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d84060140 0x7f2d84106520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:56850/0 (socket says 172.21.15.194:56850) 2024-01-18T06:55:45.421 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:45.420+0000 7f2d8259c700 1 -- 172.21.15.194:0/2924678716 learned_addr learned my addr 172.21.15.194:0/2924678716 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:55:45.421 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:45.420+0000 7f2d8259c700 1 -- 172.21.15.194:0/2924678716 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d74005040 con 0x7f2d84060140 2024-01-18T06:55:45.421 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:45.420+0000 7f2d8259c700 1 --2- 172.21.15.194:0/2924678716 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d84060140 0x7f2d84106520 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f2d74013510 tx=0x7f2d7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:45.421 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:45.420+0000 7f2d72ffd700 1 -- 172.21.15.194:0/2924678716 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d74009320 con 0x7f2d84060140 2024-01-18T06:55:45.421 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:45.420+0000 7f2d8359e700 1 -- 172.21.15.194:0/2924678716 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d84106c10 con 0x7f2d84060140 2024-01-18T06:55:45.421 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:45.420+0000 7f2d72ffd700 1 -- 172.21.15.194:0/2924678716 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2d74013f00 con 0x7f2d84060140 2024-01-18T06:55:45.421 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:45.420+0000 7f2d72ffd700 1 -- 172.21.15.194:0/2924678716 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d74014d90 con 0x7f2d84060140 2024-01-18T06:55:45.422 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:45.420+0000 7f2d8359e700 1 -- 172.21.15.194:0/2924678716 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d841070a0 con 0x7f2d84060140 2024-01-18T06:55:45.422 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:45.421+0000 7f2d72ffd700 1 -- 172.21.15.194:0/2924678716 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2d740095f0 con 0x7f2d84060140 2024-01-18T06:55:45.422 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:45.421+0000 7f2d8359e700 1 -- 172.21.15.194:0/2924678716 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d84045ad0 con 0x7f2d84060140 2024-01-18T06:55:45.423 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:45.421+0000 7f2d72ffd700 1 --2- 172.21.15.194:0/2924678716 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f2d6c02ff60 0x7f2d6c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:45.423 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:45.421+0000 7f2d72ffd700 1 -- 172.21.15.194:0/2924678716 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2d74041000 con 0x7f2d84060140 2024-01-18T06:55:45.423 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:45.422+0000 7f2d81d9b700 1 --2- 172.21.15.194:0/2924678716 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f2d6c02ff60 0x7f2d6c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:45.423 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:45.422+0000 7f2d81d9b700 1 --2- 172.21.15.194:0/2924678716 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f2d6c02ff60 0x7f2d6c032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f2d780096a0 tx=0x7f2d78006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:45.425 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:45.426+0000 7f2d72ffd700 1 -- 172.21.15.194:0/2924678716 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2d7401d7f0 con 0x7f2d84060140 2024-01-18T06:55:45.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:45 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:55:44.043281+0000 mgr.smithi047.fuhzzr (mgr.14162) 223 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:55:45.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:45.631+0000 7f2d8359e700 1 -- 172.21.15.194:0/2924678716 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2d84107f50 con 0x7f2d84060140 2024-01-18T06:55:45.631 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:45.632+0000 7f2d72ffd700 1 -- 172.21.15.194:0/2924678716 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2d74010230 con 0x7f2d84060140 2024-01-18T06:55:45.631 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:55:45.632 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:55:45.633 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:45.633+0000 7f2d8359e700 1 -- 172.21.15.194:0/2924678716 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f2d6c02ff60 msgr2=0x7f2d6c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:45.633 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:45.633+0000 7f2d8359e700 1 --2- 172.21.15.194:0/2924678716 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f2d6c02ff60 0x7f2d6c032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f2d780096a0 tx=0x7f2d78006b40).stop 2024-01-18T06:55:45.633 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:45.633+0000 7f2d8359e700 1 -- 172.21.15.194:0/2924678716 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d84060140 msgr2=0x7f2d84106520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:45.633 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:45.633+0000 7f2d8359e700 1 --2- 172.21.15.194:0/2924678716 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d84060140 0x7f2d84106520 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f2d74013510 tx=0x7f2d7400b040).stop 2024-01-18T06:55:45.633 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:45.634+0000 7f2d8359e700 1 -- 172.21.15.194:0/2924678716 shutdown_connections 2024-01-18T06:55:45.633 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:45.634+0000 7f2d8359e700 1 --2- 172.21.15.194:0/2924678716 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f2d6c02ff60 0x7f2d6c032420 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:45.633 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:45.634+0000 7f2d8359e700 1 --2- 172.21.15.194:0/2924678716 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2d84060140 0x7f2d84106520 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:45.633 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:45.634+0000 7f2d8359e700 1 -- 172.21.15.194:0/2924678716 >> 172.21.15.194:0/2924678716 conn(0x7f2d840f09a0 msgr2=0x7f2d840f14d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:45.633 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:45.634+0000 7f2d8359e700 1 -- 172.21.15.194:0/2924678716 shutdown_connections 2024-01-18T06:55:45.634 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:45.634+0000 7f2d8359e700 1 -- 172.21.15.194:0/2924678716 wait complete. 2024-01-18T06:55:45.634 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:55:46.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:46 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:55:45.632800+0000 mon.smithi047 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.194:0/2924678716' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:55:46.974 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:55:46.975 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:55:47.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:47 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:55:46.043640+0000 mgr.smithi047.fuhzzr (mgr.14162) 224 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:55:48.157 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:48.156+0000 7f0c8f9b0700 1 -- 172.21.15.194:0/3881723148 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0c880f6510 msgr2=0x7f0c880f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:48.158 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:48.156+0000 7f0c8f9b0700 1 --2- 172.21.15.194:0/3881723148 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0c880f6510 0x7f0c880f6930 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f0c84009fb0 tx=0x7f0c84009b30).stop 2024-01-18T06:55:48.158 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:48.156+0000 7f0c8f9b0700 1 -- 172.21.15.194:0/3881723148 shutdown_connections 2024-01-18T06:55:48.158 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:48.156+0000 7f0c8f9b0700 1 --2- 172.21.15.194:0/3881723148 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0c880f6510 0x7f0c880f6930 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:48.158 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:48.156+0000 7f0c8f9b0700 1 -- 172.21.15.194:0/3881723148 >> 172.21.15.194:0/3881723148 conn(0x7f0c880f1c70 msgr2=0x7f0c880f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:48.158 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:48.156+0000 7f0c8f9b0700 1 -- 172.21.15.194:0/3881723148 shutdown_connections 2024-01-18T06:55:48.158 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:48.156+0000 7f0c8f9b0700 1 -- 172.21.15.194:0/3881723148 wait complete. 2024-01-18T06:55:48.158 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:48.157+0000 7f0c8f9b0700 1 Processor -- start 2024-01-18T06:55:48.158 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:48.157+0000 7f0c8f9b0700 1 -- start start 2024-01-18T06:55:48.158 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:48.157+0000 7f0c8f9b0700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0c880f6510 0x7f0c88106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:48.158 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:48.157+0000 7f0c8f9b0700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c88106c20 con 0x7f0c880f6510 2024-01-18T06:55:48.158 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:48.157+0000 7f0c8d74c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0c880f6510 0x7f0c88106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:48.158 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:48.157+0000 7f0c8d74c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0c880f6510 0x7f0c88106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:59204/0 (socket says 172.21.15.194:59204) 2024-01-18T06:55:48.159 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:48.157+0000 7f0c8d74c700 1 -- 172.21.15.194:0/1496132504 learned_addr learned my addr 172.21.15.194:0/1496132504 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:55:48.159 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:48.158+0000 7f0c8d74c700 1 -- 172.21.15.194:0/1496132504 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c84005040 con 0x7f0c880f6510 2024-01-18T06:55:48.159 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:48.158+0000 7f0c8d74c700 1 --2- 172.21.15.194:0/1496132504 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0c880f6510 0x7f0c88106730 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f0c84014000 tx=0x7f0c8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:48.159 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:48.158+0000 7f0c7dffb700 1 -- 172.21.15.194:0/1496132504 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c840095f0 con 0x7f0c880f6510 2024-01-18T06:55:48.159 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:48.158+0000 7f0c7dffb700 1 -- 172.21.15.194:0/1496132504 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0c84007e70 con 0x7f0c880f6510 2024-01-18T06:55:48.159 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:48.158+0000 7f0c8f9b0700 1 -- 172.21.15.194:0/1496132504 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c88106e20 con 0x7f0c880f6510 2024-01-18T06:55:48.159 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:48.158+0000 7f0c7dffb700 1 -- 172.21.15.194:0/1496132504 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c84014ea0 con 0x7f0c880f6510 2024-01-18T06:55:48.159 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:48.158+0000 7f0c8f9b0700 1 -- 172.21.15.194:0/1496132504 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c881072b0 con 0x7f0c880f6510 2024-01-18T06:55:48.160 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:48.159+0000 7f0c7dffb700 1 -- 172.21.15.194:0/1496132504 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0c84025070 con 0x7f0c880f6510 2024-01-18T06:55:48.160 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:48.159+0000 7f0c8f9b0700 1 -- 172.21.15.194:0/1496132504 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c88045ad0 con 0x7f0c880f6510 2024-01-18T06:55:48.160 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:48.159+0000 7f0c7dffb700 1 --2- 172.21.15.194:0/1496132504 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f0c7402ffb0 0x7f0c74032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:48.160 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:48.159+0000 7f0c7dffb700 1 -- 172.21.15.194:0/1496132504 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0c84040950 con 0x7f0c880f6510 2024-01-18T06:55:48.160 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:48.159+0000 7f0c8cf4b700 1 --2- 172.21.15.194:0/1496132504 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f0c7402ffb0 0x7f0c74032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:48.160 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:48.160+0000 7f0c8cf4b700 1 --2- 172.21.15.194:0/1496132504 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f0c7402ffb0 0x7f0c74032470 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f0c780096a0 tx=0x7f0c78006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:48.163 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:48.164+0000 7f0c7dffb700 1 -- 172.21.15.194:0/1496132504 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0c840079c0 con 0x7f0c880f6510 2024-01-18T06:55:48.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:48.372+0000 7f0c8f9b0700 1 -- 172.21.15.194:0/1496132504 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0c880fbf60 con 0x7f0c880f6510 2024-01-18T06:55:48.372 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:48.372+0000 7f0c7dffb700 1 -- 172.21.15.194:0/1496132504 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0c8400f300 con 0x7f0c880f6510 2024-01-18T06:55:48.372 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:55:48.372 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:55:48.374 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:48.374+0000 7f0c8f9b0700 1 -- 172.21.15.194:0/1496132504 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f0c7402ffb0 msgr2=0x7f0c74032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:48.374 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:48.374+0000 7f0c8f9b0700 1 --2- 172.21.15.194:0/1496132504 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f0c7402ffb0 0x7f0c74032470 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f0c780096a0 tx=0x7f0c78006b40).stop 2024-01-18T06:55:48.374 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:48.374+0000 7f0c8f9b0700 1 -- 172.21.15.194:0/1496132504 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0c880f6510 msgr2=0x7f0c88106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:48.374 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:48.374+0000 7f0c8f9b0700 1 --2- 172.21.15.194:0/1496132504 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0c880f6510 0x7f0c88106730 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f0c84014000 tx=0x7f0c8400b040).stop 2024-01-18T06:55:48.374 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:48.374+0000 7f0c8f9b0700 1 -- 172.21.15.194:0/1496132504 shutdown_connections 2024-01-18T06:55:48.374 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:48.374+0000 7f0c8f9b0700 1 --2- 172.21.15.194:0/1496132504 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f0c7402ffb0 0x7f0c74032470 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:48.374 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:48.374+0000 7f0c8f9b0700 1 --2- 172.21.15.194:0/1496132504 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0c880f6510 0x7f0c88106730 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:48.374 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:48.374+0000 7f0c8f9b0700 1 -- 172.21.15.194:0/1496132504 >> 172.21.15.194:0/1496132504 conn(0x7f0c880f1c70 msgr2=0x7f0c880f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:48.374 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:48.374+0000 7f0c8f9b0700 1 -- 172.21.15.194:0/1496132504 shutdown_connections 2024-01-18T06:55:48.374 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:48.374+0000 7f0c8f9b0700 1 -- 172.21.15.194:0/1496132504 wait complete. 2024-01-18T06:55:48.375 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:55:49.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:48 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:55:48.044031+0000 mgr.smithi047.fuhzzr (mgr.14162) 225 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:55:49.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:48 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:55:48.373381+0000 mon.smithi047 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.194:0/1496132504' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:55:49.764 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:55:49.764 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:55:50.960 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:50.958+0000 7f8897f58700 1 -- 172.21.15.194:0/1025765706 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88900f6510 msgr2=0x7f88900f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:50.960 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:50.958+0000 7f8897f58700 1 --2- 172.21.15.194:0/1025765706 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88900f6510 0x7f88900f6930 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f8880009fb0 tx=0x7f8880009b30).stop 2024-01-18T06:55:50.960 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:50.958+0000 7f8897f58700 1 -- 172.21.15.194:0/1025765706 shutdown_connections 2024-01-18T06:55:50.960 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:50.958+0000 7f8897f58700 1 --2- 172.21.15.194:0/1025765706 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88900f6510 0x7f88900f6930 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:50.960 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:50.958+0000 7f8897f58700 1 -- 172.21.15.194:0/1025765706 >> 172.21.15.194:0/1025765706 conn(0x7f88900f1c70 msgr2=0x7f88900f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:50.960 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:50.958+0000 7f8897f58700 1 -- 172.21.15.194:0/1025765706 shutdown_connections 2024-01-18T06:55:50.960 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:50.958+0000 7f8897f58700 1 -- 172.21.15.194:0/1025765706 wait complete. 2024-01-18T06:55:50.960 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:50.959+0000 7f8897f58700 1 Processor -- start 2024-01-18T06:55:50.960 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:50.959+0000 7f8897f58700 1 -- start start 2024-01-18T06:55:50.961 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:50.959+0000 7f8897f58700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88900f6510 0x7f8890106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:50.961 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:50.959+0000 7f8897f58700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8890106c20 con 0x7f88900f6510 2024-01-18T06:55:50.961 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:50.959+0000 7f8895cf4700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88900f6510 0x7f8890106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:50.961 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:50.959+0000 7f8895cf4700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88900f6510 0x7f8890106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:59230/0 (socket says 172.21.15.194:59230) 2024-01-18T06:55:50.961 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:50.959+0000 7f8895cf4700 1 -- 172.21.15.194:0/2625712176 learned_addr learned my addr 172.21.15.194:0/2625712176 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:55:50.961 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:50.960+0000 7f8895cf4700 1 -- 172.21.15.194:0/2625712176 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8880005040 con 0x7f88900f6510 2024-01-18T06:55:50.961 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:50.960+0000 7f8895cf4700 1 --2- 172.21.15.194:0/2625712176 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88900f6510 0x7f8890106730 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f8880013510 tx=0x7f888000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:50.961 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:50.960+0000 7f88867fc700 1 -- 172.21.15.194:0/2625712176 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8880009320 con 0x7f88900f6510 2024-01-18T06:55:50.961 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:50.960+0000 7f8897f58700 1 -- 172.21.15.194:0/2625712176 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8890106e20 con 0x7f88900f6510 2024-01-18T06:55:50.961 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:50.960+0000 7f88867fc700 1 -- 172.21.15.194:0/2625712176 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8880013f00 con 0x7f88900f6510 2024-01-18T06:55:50.961 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:50.960+0000 7f88867fc700 1 -- 172.21.15.194:0/2625712176 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8880014d90 con 0x7f88900f6510 2024-01-18T06:55:50.961 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:50.960+0000 7f8897f58700 1 -- 172.21.15.194:0/2625712176 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88901072b0 con 0x7f88900f6510 2024-01-18T06:55:50.962 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:50.961+0000 7f88867fc700 1 -- 172.21.15.194:0/2625712176 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f88800095f0 con 0x7f88900f6510 2024-01-18T06:55:50.962 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:50.961+0000 7f8897f58700 1 -- 172.21.15.194:0/2625712176 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8890045ad0 con 0x7f88900f6510 2024-01-18T06:55:50.962 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:50.961+0000 7f88867fc700 1 --2- 172.21.15.194:0/2625712176 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f887c02ff60 0x7f887c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:50.962 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:50.961+0000 7f88867fc700 1 -- 172.21.15.194:0/2625712176 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f88800410c0 con 0x7f88900f6510 2024-01-18T06:55:50.962 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:50.961+0000 7f88954f3700 1 --2- 172.21.15.194:0/2625712176 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f887c02ff60 0x7f887c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:50.963 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:50.962+0000 7f88954f3700 1 --2- 172.21.15.194:0/2625712176 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f887c02ff60 0x7f887c032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f888c0096a0 tx=0x7f888c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:50.965 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:50.966+0000 7f88867fc700 1 -- 172.21.15.194:0/2625712176 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8880014420 con 0x7f88900f6510 2024-01-18T06:55:51.173 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:51.174+0000 7f8897f58700 1 -- 172.21.15.194:0/2625712176 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f88900fbf60 con 0x7f88900f6510 2024-01-18T06:55:51.174 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:51.174+0000 7f88867fc700 1 -- 172.21.15.194:0/2625712176 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8880010230 con 0x7f88900f6510 2024-01-18T06:55:51.174 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:55:51.175 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:55:51.175 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:51.176+0000 7f8897f58700 1 -- 172.21.15.194:0/2625712176 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f887c02ff60 msgr2=0x7f887c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:51.175 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:51.176+0000 7f8897f58700 1 --2- 172.21.15.194:0/2625712176 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f887c02ff60 0x7f887c032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f888c0096a0 tx=0x7f888c006b40).stop 2024-01-18T06:55:51.176 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:51.176+0000 7f8897f58700 1 -- 172.21.15.194:0/2625712176 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88900f6510 msgr2=0x7f8890106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:51.176 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:51.176+0000 7f8897f58700 1 --2- 172.21.15.194:0/2625712176 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88900f6510 0x7f8890106730 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f8880013510 tx=0x7f888000b040).stop 2024-01-18T06:55:51.176 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:51.176+0000 7f8897f58700 1 -- 172.21.15.194:0/2625712176 shutdown_connections 2024-01-18T06:55:51.176 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:51.176+0000 7f8897f58700 1 --2- 172.21.15.194:0/2625712176 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f887c02ff60 0x7f887c032420 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:51.176 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:51.176+0000 7f8897f58700 1 --2- 172.21.15.194:0/2625712176 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88900f6510 0x7f8890106730 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:51.176 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:51.176+0000 7f8897f58700 1 -- 172.21.15.194:0/2625712176 >> 172.21.15.194:0/2625712176 conn(0x7f88900f1c70 msgr2=0x7f88900f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:51.176 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:51.176+0000 7f8897f58700 1 -- 172.21.15.194:0/2625712176 shutdown_connections 2024-01-18T06:55:51.176 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:51.176+0000 7f8897f58700 1 -- 172.21.15.194:0/2625712176 wait complete. 2024-01-18T06:55:51.176 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:55:51.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:51 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:55:50.044385+0000 mgr.smithi047.fuhzzr (mgr.14162) 226 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:55:52.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:52 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:55:51.175396+0000 mon.smithi047 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.194:0/2625712176' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:55:52.567 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:55:52.567 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:55:53.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:53 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:55:52.044691+0000 mgr.smithi047.fuhzzr (mgr.14162) 227 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:55:53.734 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:53.733+0000 7fbee1370700 1 -- 172.21.15.194:0/1722510549 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbedc0f62c0 msgr2=0x7fbedc0f66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:53.734 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:53.733+0000 7fbee1370700 1 --2- 172.21.15.194:0/1722510549 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbedc0f62c0 0x7fbedc0f66e0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7fbecc009fb0 tx=0x7fbecc009b30).stop 2024-01-18T06:55:53.734 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:53.733+0000 7fbee1370700 1 -- 172.21.15.194:0/1722510549 shutdown_connections 2024-01-18T06:55:53.734 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:53.733+0000 7fbee1370700 1 --2- 172.21.15.194:0/1722510549 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbedc0f62c0 0x7fbedc0f66e0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:53.734 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:53.733+0000 7fbee1370700 1 -- 172.21.15.194:0/1722510549 >> 172.21.15.194:0/1722510549 conn(0x7fbedc0f1a20 msgr2=0x7fbedc0f3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:53.734 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:53.733+0000 7fbee1370700 1 -- 172.21.15.194:0/1722510549 shutdown_connections 2024-01-18T06:55:53.735 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:53.733+0000 7fbee1370700 1 -- 172.21.15.194:0/1722510549 wait complete. 2024-01-18T06:55:53.735 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:53.734+0000 7fbee1370700 1 Processor -- start 2024-01-18T06:55:53.735 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:53.734+0000 7fbee1370700 1 -- start start 2024-01-18T06:55:53.735 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:53.734+0000 7fbee1370700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbedc0f62c0 0x7fbedc06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:53.735 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:53.734+0000 7fbee1370700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbedc069e20 con 0x7fbedc0f62c0 2024-01-18T06:55:53.735 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:53.734+0000 7fbedaffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbedc0f62c0 0x7fbedc06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:53.735 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:53.734+0000 7fbedaffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbedc0f62c0 0x7fbedc06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:59246/0 (socket says 172.21.15.194:59246) 2024-01-18T06:55:53.735 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:53.734+0000 7fbedaffd700 1 -- 172.21.15.194:0/3925292243 learned_addr learned my addr 172.21.15.194:0/3925292243 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:55:53.736 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:53.735+0000 7fbedaffd700 1 -- 172.21.15.194:0/3925292243 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbecc005040 con 0x7fbedc0f62c0 2024-01-18T06:55:53.736 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:53.735+0000 7fbedaffd700 1 --2- 172.21.15.194:0/3925292243 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbedc0f62c0 0x7fbedc06b9a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fbecc013510 tx=0x7fbecc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:53.737 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:53.735+0000 7fbec37fe700 1 -- 172.21.15.194:0/3925292243 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbecc009320 con 0x7fbedc0f62c0 2024-01-18T06:55:53.737 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:53.735+0000 7fbee1370700 1 -- 172.21.15.194:0/3925292243 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbedc069fc0 con 0x7fbedc0f62c0 2024-01-18T06:55:53.737 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:53.735+0000 7fbec37fe700 1 -- 172.21.15.194:0/3925292243 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbecc013f00 con 0x7fbedc0f62c0 2024-01-18T06:55:53.737 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:53.735+0000 7fbec37fe700 1 -- 172.21.15.194:0/3925292243 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbecc014d90 con 0x7fbedc0f62c0 2024-01-18T06:55:53.737 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:53.735+0000 7fbee1370700 1 -- 172.21.15.194:0/3925292243 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbedc06a450 con 0x7fbedc0f62c0 2024-01-18T06:55:53.737 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:53.736+0000 7fbec37fe700 1 -- 172.21.15.194:0/3925292243 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbecc0095f0 con 0x7fbedc0f62c0 2024-01-18T06:55:53.738 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:53.736+0000 7fbee1370700 1 -- 172.21.15.194:0/3925292243 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbedc059100 con 0x7fbedc0f62c0 2024-01-18T06:55:53.738 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:53.736+0000 7fbec37fe700 1 --2- 172.21.15.194:0/3925292243 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fbec402ff60 0x7fbec4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:53.738 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:53.736+0000 7fbec37fe700 1 -- 172.21.15.194:0/3925292243 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbecc0410c0 con 0x7fbedc0f62c0 2024-01-18T06:55:53.738 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:53.736+0000 7fbeda7fc700 1 --2- 172.21.15.194:0/3925292243 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fbec402ff60 0x7fbec4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:53.738 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:53.737+0000 7fbeda7fc700 1 --2- 172.21.15.194:0/3925292243 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fbec402ff60 0x7fbec4032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fbed00096a0 tx=0x7fbed0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:53.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:53.741+0000 7fbec37fe700 1 -- 172.21.15.194:0/3925292243 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbecc014420 con 0x7fbedc0f62c0 2024-01-18T06:55:53.947 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:53.948+0000 7fbee1370700 1 -- 172.21.15.194:0/3925292243 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbedc045ad0 con 0x7fbedc0f62c0 2024-01-18T06:55:53.948 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:53.949+0000 7fbec37fe700 1 -- 172.21.15.194:0/3925292243 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbecc010230 con 0x7fbedc0f62c0 2024-01-18T06:55:53.949 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:55:53.949 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:55:53.951 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:53.950+0000 7fbee1370700 1 -- 172.21.15.194:0/3925292243 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fbec402ff60 msgr2=0x7fbec4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:53.951 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:53.950+0000 7fbee1370700 1 --2- 172.21.15.194:0/3925292243 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fbec402ff60 0x7fbec4032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fbed00096a0 tx=0x7fbed0006b40).stop 2024-01-18T06:55:53.951 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:53.950+0000 7fbee1370700 1 -- 172.21.15.194:0/3925292243 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbedc0f62c0 msgr2=0x7fbedc06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:53.951 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:53.950+0000 7fbee1370700 1 --2- 172.21.15.194:0/3925292243 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbedc0f62c0 0x7fbedc06b9a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fbecc013510 tx=0x7fbecc00b040).stop 2024-01-18T06:55:53.951 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:53.950+0000 7fbee1370700 1 -- 172.21.15.194:0/3925292243 shutdown_connections 2024-01-18T06:55:53.951 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:53.950+0000 7fbee1370700 1 --2- 172.21.15.194:0/3925292243 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fbec402ff60 0x7fbec4032420 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:53.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:53.950+0000 7fbee1370700 1 --2- 172.21.15.194:0/3925292243 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbedc0f62c0 0x7fbedc06b9a0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:53.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:53.950+0000 7fbee1370700 1 -- 172.21.15.194:0/3925292243 >> 172.21.15.194:0/3925292243 conn(0x7fbedc0f1a20 msgr2=0x7fbedc0f2580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:53.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:53.950+0000 7fbee1370700 1 -- 172.21.15.194:0/3925292243 shutdown_connections 2024-01-18T06:55:53.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:53.950+0000 7fbee1370700 1 -- 172.21.15.194:0/3925292243 wait complete. 2024-01-18T06:55:53.952 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:55:54.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:54 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:55:53.949745+0000 mon.smithi047 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.194:0/3925292243' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:55:55.329 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:55:55.329 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:55:55.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:55 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:55:54.044948+0000 mgr.smithi047.fuhzzr (mgr.14162) 228 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:55:56.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cephadm 2024-01-18T06:55:55.005298+0000 mgr.smithi047.fuhzzr (mgr.14162) 229 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi047 ... 2024-01-18T06:55:56.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: Verifying port 9100 ... 2024-01-18T06:55:56.464 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: Non-zero exit code 1 from systemctl start ceph-653e4012-b5cd-11ee-95ae-87774f69a715@node-exporter.smithi047 2024-01-18T06:55:56.464 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: systemctl: stderr Job for ceph-653e4012-b5cd-11ee-95ae-87774f69a715@node-exporter.smithi047.service failed because a timeout was exceeded. 2024-01-18T06:55:56.464 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: systemctl: stderr See "systemctl status ceph-653e4012-b5cd-11ee-95ae-87774f69a715@node-exporter.smithi047.service" and "journalctl -xe" for details. 2024-01-18T06:55:56.464 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: Traceback (most recent call last): 2024-01-18T06:55:56.464 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: File "/var/lib/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-18T06:55:56.464 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: main() 2024-01-18T06:55:56.464 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: File "/var/lib/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-18T06:55:56.464 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: r = ctx.func(ctx) 2024-01-18T06:55:56.464 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: File "/var/lib/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-18T06:55:56.464 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: return func(ctx) 2024-01-18T06:55:56.464 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: File "/var/lib/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-18T06:55:56.464 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: ports=daemon_ports) 2024-01-18T06:55:56.464 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: File "/var/lib/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-01-18T06:55:56.465 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: c, osd_fsid=osd_fsid, ports=ports) 2024-01-18T06:55:56.465 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: File "/var/lib/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-01-18T06:55:56.465 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-01-18T06:55:56.465 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: File "/var/lib/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-18T06:55:56.465 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-18T06:55:56.465 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: RuntimeError: Failed command: systemctl start ceph-653e4012-b5cd-11ee-95ae-87774f69a715@node-exporter.smithi047 2024-01-18T06:55:56.465 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: Traceback (most recent call last): 2024-01-18T06:55:56.465 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-18T06:55:56.465 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: yield (conn, connr) 2024-01-18T06:55:56.465 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-18T06:55:56.465 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: code, '\n'.join(err))) 2024-01-18T06:55:56.465 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi047 ... 2024-01-18T06:55:56.465 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: Verifying port 9100 ... 2024-01-18T06:55:56.465 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: Non-zero exit code 1 from systemctl start ceph-653e4012-b5cd-11ee-95ae-87774f69a715@node-exporter.smithi047 2024-01-18T06:55:56.466 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: systemctl: stderr Job for ceph-653e4012-b5cd-11ee-95ae-87774f69a715@node-exporter.smithi047.service failed because a timeout was exceeded. 2024-01-18T06:55:56.466 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: systemctl: stderr See "systemctl status ceph-653e4012-b5cd-11ee-95ae-87774f69a715@node-exporter.smithi047.service" and "journalctl -xe" for details. 2024-01-18T06:55:56.466 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: Traceback (most recent call last): 2024-01-18T06:55:56.466 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: File "/var/lib/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-18T06:55:56.466 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: main() 2024-01-18T06:55:56.466 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: File "/var/lib/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-18T06:55:56.466 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: r = ctx.func(ctx) 2024-01-18T06:55:56.466 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: File "/var/lib/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-18T06:55:56.466 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: return func(ctx) 2024-01-18T06:55:56.466 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: File "/var/lib/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-18T06:55:56.466 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: ports=daemon_ports) 2024-01-18T06:55:56.466 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: File "/var/lib/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-01-18T06:55:56.466 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: c, osd_fsid=osd_fsid, ports=ports) 2024-01-18T06:55:56.466 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: File "/var/lib/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-01-18T06:55:56.467 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-01-18T06:55:56.467 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: File "/var/lib/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-18T06:55:56.467 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-18T06:55:56.467 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: RuntimeError: Failed command: systemctl start ceph-653e4012-b5cd-11ee-95ae-87774f69a715@node-exporter.smithi047 2024-01-18T06:55:56.467 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:56 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cephadm 2024-01-18T06:55:55.010898+0000 mgr.smithi047.fuhzzr (mgr.14162) 230 : cephadm [INF] Deploying daemon prometheus.smithi047 on smithi047 2024-01-18T06:55:56.523 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:56.523+0000 7fb4028cb700 1 -- 172.21.15.194:0/591805917 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3fc0f62a0 msgr2=0x7fb3fc0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:56.523 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:56.524+0000 7fb4028cb700 1 --2- 172.21.15.194:0/591805917 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3fc0f62a0 0x7fb3fc0f66c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7fb3ec009fb0 tx=0x7fb3ec009b30).stop 2024-01-18T06:55:56.523 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:56.524+0000 7fb4028cb700 1 -- 172.21.15.194:0/591805917 shutdown_connections 2024-01-18T06:55:56.523 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:56.524+0000 7fb4028cb700 1 --2- 172.21.15.194:0/591805917 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3fc0f62a0 0x7fb3fc0f66c0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:56.524 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:56.524+0000 7fb4028cb700 1 -- 172.21.15.194:0/591805917 >> 172.21.15.194:0/591805917 conn(0x7fb3fc0f1ac0 msgr2=0x7fb3fc0f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:56.524 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:56.524+0000 7fb4028cb700 1 -- 172.21.15.194:0/591805917 shutdown_connections 2024-01-18T06:55:56.524 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:56.524+0000 7fb4028cb700 1 -- 172.21.15.194:0/591805917 wait complete. 2024-01-18T06:55:56.525 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:56.524+0000 7fb4028cb700 1 Processor -- start 2024-01-18T06:55:56.525 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:56.524+0000 7fb4028cb700 1 -- start start 2024-01-18T06:55:56.525 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:56.525+0000 7fb4028cb700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3fc0f62a0 0x7fb3fc1064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:56.525 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:56.525+0000 7fb4028cb700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc106990 con 0x7fb3fc0f62a0 2024-01-18T06:55:56.525 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:56.525+0000 7fb4018c9700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3fc0f62a0 0x7fb3fc1064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:56.526 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:56.525+0000 7fb4018c9700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3fc0f62a0 0x7fb3fc1064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:59266/0 (socket says 172.21.15.194:59266) 2024-01-18T06:55:56.526 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:56.525+0000 7fb4018c9700 1 -- 172.21.15.194:0/172875555 learned_addr learned my addr 172.21.15.194:0/172875555 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:55:56.526 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:56.525+0000 7fb4018c9700 1 -- 172.21.15.194:0/172875555 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3ec005040 con 0x7fb3fc0f62a0 2024-01-18T06:55:56.526 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:56.525+0000 7fb4018c9700 1 --2- 172.21.15.194:0/172875555 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3fc0f62a0 0x7fb3fc1064a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7fb3ec013510 tx=0x7fb3ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:56.526 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:56.526+0000 7fb3f27fc700 1 -- 172.21.15.194:0/172875555 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3ec009320 con 0x7fb3fc0f62a0 2024-01-18T06:55:56.526 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:56.526+0000 7fb4028cb700 1 -- 172.21.15.194:0/172875555 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3fc106b90 con 0x7fb3fc0f62a0 2024-01-18T06:55:56.526 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:56.526+0000 7fb3f27fc700 1 -- 172.21.15.194:0/172875555 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb3ec013f00 con 0x7fb3fc0f62a0 2024-01-18T06:55:56.526 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:56.526+0000 7fb3f27fc700 1 -- 172.21.15.194:0/172875555 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3ec014dc0 con 0x7fb3fc0f62a0 2024-01-18T06:55:56.526 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:56.526+0000 7fb4028cb700 1 -- 172.21.15.194:0/172875555 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3fc107020 con 0x7fb3fc0f62a0 2024-01-18T06:55:56.527 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:56.526+0000 7fb3f27fc700 1 -- 172.21.15.194:0/172875555 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb3ec01d3d0 con 0x7fb3fc0f62a0 2024-01-18T06:55:56.527 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:56.527+0000 7fb3f27fc700 1 --2- 172.21.15.194:0/172875555 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb3e802ff60 0x7fb3e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:56.527 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:56.527+0000 7fb3f27fc700 1 -- 172.21.15.194:0/172875555 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb3ec041350 con 0x7fb3fc0f62a0 2024-01-18T06:55:56.527 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:56.527+0000 7fb4028cb700 1 -- 172.21.15.194:0/172875555 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3fc045ad0 con 0x7fb3fc0f62a0 2024-01-18T06:55:56.527 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:56.527+0000 7fb4010c8700 1 --2- 172.21.15.194:0/172875555 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb3e802ff60 0x7fb3e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:56.528 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:56.527+0000 7fb4010c8700 1 --2- 172.21.15.194:0/172875555 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb3e802ff60 0x7fb3e8032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fb3f80096a0 tx=0x7fb3f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:56.530 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:56.531+0000 7fb3f27fc700 1 -- 172.21.15.194:0/172875555 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb3ec012920 con 0x7fb3fc0f62a0 2024-01-18T06:55:56.736 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:56.737+0000 7fb4028cb700 1 -- 172.21.15.194:0/172875555 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb3fc107ed0 con 0x7fb3fc0f62a0 2024-01-18T06:55:56.737 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:56.737+0000 7fb3f27fc700 1 -- 172.21.15.194:0/172875555 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb3ec02b030 con 0x7fb3fc0f62a0 2024-01-18T06:55:56.737 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:55:56.737 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:55:56.738 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:56.739+0000 7fb4028cb700 1 -- 172.21.15.194:0/172875555 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb3e802ff60 msgr2=0x7fb3e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:56.739 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:56.739+0000 7fb4028cb700 1 --2- 172.21.15.194:0/172875555 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb3e802ff60 0x7fb3e8032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fb3f80096a0 tx=0x7fb3f8006b40).stop 2024-01-18T06:55:56.739 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:56.739+0000 7fb4028cb700 1 -- 172.21.15.194:0/172875555 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3fc0f62a0 msgr2=0x7fb3fc1064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:56.739 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:56.739+0000 7fb4028cb700 1 --2- 172.21.15.194:0/172875555 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3fc0f62a0 0x7fb3fc1064a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7fb3ec013510 tx=0x7fb3ec00b040).stop 2024-01-18T06:55:56.739 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:56.739+0000 7fb4028cb700 1 -- 172.21.15.194:0/172875555 shutdown_connections 2024-01-18T06:55:56.739 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:56.739+0000 7fb4028cb700 1 --2- 172.21.15.194:0/172875555 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb3e802ff60 0x7fb3e8032420 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:56.739 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:56.739+0000 7fb4028cb700 1 --2- 172.21.15.194:0/172875555 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3fc0f62a0 0x7fb3fc1064a0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:56.739 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:56.739+0000 7fb4028cb700 1 -- 172.21.15.194:0/172875555 >> 172.21.15.194:0/172875555 conn(0x7fb3fc0f1ac0 msgr2=0x7fb3fc0f2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:56.739 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:56.739+0000 7fb4028cb700 1 -- 172.21.15.194:0/172875555 shutdown_connections 2024-01-18T06:55:56.739 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:56.739+0000 7fb4028cb700 1 -- 172.21.15.194:0/172875555 wait complete. 2024-01-18T06:55:56.739 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:55:57.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:57 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:55:56.045298+0000 mgr.smithi047.fuhzzr (mgr.14162) 231 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:55:57.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:57 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:55:56.738359+0000 mon.smithi047 (mon.0) 288 : audit [DBG] from='client.? 172.21.15.194:0/172875555' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:55:58.133 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:55:58.133 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:55:59.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:59.272+0000 7fda40fd2700 1 -- 172.21.15.194:0/1354514974 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda3c0f4b00 msgr2=0x7fda3c0f4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:59.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:59.272+0000 7fda40fd2700 1 --2- 172.21.15.194:0/1354514974 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda3c0f4b00 0x7fda3c0f4f20 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7fda30009fb0 tx=0x7fda30009b30).stop 2024-01-18T06:55:59.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:59.272+0000 7fda40fd2700 1 -- 172.21.15.194:0/1354514974 shutdown_connections 2024-01-18T06:55:59.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:59.272+0000 7fda40fd2700 1 --2- 172.21.15.194:0/1354514974 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda3c0f4b00 0x7fda3c0f4f20 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:59.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:59.272+0000 7fda40fd2700 1 -- 172.21.15.194:0/1354514974 >> 172.21.15.194:0/1354514974 conn(0x7fda3c0f0260 msgr2=0x7fda3c0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:59.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:59.272+0000 7fda40fd2700 1 -- 172.21.15.194:0/1354514974 shutdown_connections 2024-01-18T06:55:59.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:59.272+0000 7fda40fd2700 1 -- 172.21.15.194:0/1354514974 wait complete. 2024-01-18T06:55:59.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:59.273+0000 7fda40fd2700 1 Processor -- start 2024-01-18T06:55:59.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:59.273+0000 7fda40fd2700 1 -- start start 2024-01-18T06:55:59.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:59.273+0000 7fda40fd2700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda3c0f4b00 0x7fda3c108910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:59.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:59.273+0000 7fda40fd2700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c108e00 con 0x7fda3c0f4b00 2024-01-18T06:55:59.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:59.273+0000 7fda3a59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda3c0f4b00 0x7fda3c108910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:59.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:59.273+0000 7fda3a59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda3c0f4b00 0x7fda3c108910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:59494/0 (socket says 172.21.15.194:59494) 2024-01-18T06:55:59.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:59.273+0000 7fda3a59c700 1 -- 172.21.15.194:0/2175803001 learned_addr learned my addr 172.21.15.194:0/2175803001 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:55:59.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:59.273+0000 7fda3a59c700 1 -- 172.21.15.194:0/2175803001 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda30005040 con 0x7fda3c0f4b00 2024-01-18T06:55:59.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:59.273+0000 7fda3a59c700 1 --2- 172.21.15.194:0/2175803001 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda3c0f4b00 0x7fda3c108910 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7fda30013510 tx=0x7fda3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:59.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:59.274+0000 7fda2affd700 1 -- 172.21.15.194:0/2175803001 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda30009320 con 0x7fda3c0f4b00 2024-01-18T06:55:59.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:59.274+0000 7fda40fd2700 1 -- 172.21.15.194:0/2175803001 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda3c109000 con 0x7fda3c0f4b00 2024-01-18T06:55:59.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:59.274+0000 7fda2affd700 1 -- 172.21.15.194:0/2175803001 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fda30013f00 con 0x7fda3c0f4b00 2024-01-18T06:55:59.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:59.274+0000 7fda2affd700 1 -- 172.21.15.194:0/2175803001 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda30014d90 con 0x7fda3c0f4b00 2024-01-18T06:55:59.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:59.274+0000 7fda40fd2700 1 -- 172.21.15.194:0/2175803001 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda3c109490 con 0x7fda3c0f4b00 2024-01-18T06:55:59.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:59.274+0000 7fda40fd2700 1 -- 172.21.15.194:0/2175803001 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda3c045ad0 con 0x7fda3c0f4b00 2024-01-18T06:55:59.276 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:59.274+0000 7fda2affd700 1 -- 172.21.15.194:0/2175803001 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fda30014420 con 0x7fda3c0f4b00 2024-01-18T06:55:59.276 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:59.275+0000 7fda2affd700 1 --2- 172.21.15.194:0/2175803001 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fda2402ff60 0x7fda24032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:55:59.276 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:59.275+0000 7fda2affd700 1 -- 172.21.15.194:0/2175803001 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fda30040ff0 con 0x7fda3c0f4b00 2024-01-18T06:55:59.276 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:59.275+0000 7fda39d9b700 1 --2- 172.21.15.194:0/2175803001 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fda2402ff60 0x7fda24032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:55:59.276 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:59.275+0000 7fda39d9b700 1 --2- 172.21.15.194:0/2175803001 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fda2402ff60 0x7fda24032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fda2c0096a0 tx=0x7fda2c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:55:59.278 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:59.279+0000 7fda2affd700 1 -- 172.21.15.194:0/2175803001 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fda300095f0 con 0x7fda3c0f4b00 2024-01-18T06:55:59.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:58 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:55:58.045686+0000 mgr.smithi047.fuhzzr (mgr.14162) 232 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:55:59.486 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:59.487+0000 7fda40fd2700 1 -- 172.21.15.194:0/2175803001 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fda3c0fa5c0 con 0x7fda3c0f4b00 2024-01-18T06:55:59.487 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:59.487+0000 7fda2affd700 1 -- 172.21.15.194:0/2175803001 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fda3001db70 con 0x7fda3c0f4b00 2024-01-18T06:55:59.487 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:55:59.487 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:55:59.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:59.489+0000 7fda40fd2700 1 -- 172.21.15.194:0/2175803001 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fda2402ff60 msgr2=0x7fda24032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:59.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:59.489+0000 7fda40fd2700 1 --2- 172.21.15.194:0/2175803001 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fda2402ff60 0x7fda24032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fda2c0096a0 tx=0x7fda2c006b40).stop 2024-01-18T06:55:59.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:59.489+0000 7fda40fd2700 1 -- 172.21.15.194:0/2175803001 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda3c0f4b00 msgr2=0x7fda3c108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:55:59.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:59.489+0000 7fda40fd2700 1 --2- 172.21.15.194:0/2175803001 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda3c0f4b00 0x7fda3c108910 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7fda30013510 tx=0x7fda3000b040).stop 2024-01-18T06:55:59.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:59.489+0000 7fda40fd2700 1 -- 172.21.15.194:0/2175803001 shutdown_connections 2024-01-18T06:55:59.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:59.489+0000 7fda40fd2700 1 --2- 172.21.15.194:0/2175803001 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fda2402ff60 0x7fda24032420 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:59.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:59.489+0000 7fda40fd2700 1 --2- 172.21.15.194:0/2175803001 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fda3c0f4b00 0x7fda3c108910 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:55:59.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:59.489+0000 7fda40fd2700 1 -- 172.21.15.194:0/2175803001 >> 172.21.15.194:0/2175803001 conn(0x7fda3c0f0260 msgr2=0x7fda3c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:55:59.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:59.489+0000 7fda40fd2700 1 -- 172.21.15.194:0/2175803001 shutdown_connections 2024-01-18T06:55:59.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:55:59.489+0000 7fda40fd2700 1 -- 172.21.15.194:0/2175803001 wait complete. 2024-01-18T06:55:59.490 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:56:00.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:55:59 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:55:59.488421+0000 mon.smithi047 (mon.0) 289 : audit [DBG] from='client.? 172.21.15.194:0/2175803001' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:56:00.859 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:56:00.859 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:56:01.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:00 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:56:00.046009+0000 mgr.smithi047.fuhzzr (mgr.14162) 233 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:56:02.017 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:02.017+0000 7fc383cce700 1 -- 172.21.15.194:0/889587501 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc37c0f6510 msgr2=0x7fc37c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:02.017 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:02.017+0000 7fc383cce700 1 --2- 172.21.15.194:0/889587501 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc37c0f6510 0x7fc37c0f6930 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7fc36c009fb0 tx=0x7fc36c009b30).stop 2024-01-18T06:56:02.017 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:02.017+0000 7fc383cce700 1 -- 172.21.15.194:0/889587501 shutdown_connections 2024-01-18T06:56:02.017 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:02.017+0000 7fc383cce700 1 --2- 172.21.15.194:0/889587501 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc37c0f6510 0x7fc37c0f6930 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:02.017 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:02.017+0000 7fc383cce700 1 -- 172.21.15.194:0/889587501 >> 172.21.15.194:0/889587501 conn(0x7fc37c0f1c70 msgr2=0x7fc37c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:56:02.017 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:02.017+0000 7fc383cce700 1 -- 172.21.15.194:0/889587501 shutdown_connections 2024-01-18T06:56:02.017 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:02.017+0000 7fc383cce700 1 -- 172.21.15.194:0/889587501 wait complete. 2024-01-18T06:56:02.019 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:02.018+0000 7fc383cce700 1 Processor -- start 2024-01-18T06:56:02.019 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:02.018+0000 7fc383cce700 1 -- start start 2024-01-18T06:56:02.019 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:02.018+0000 7fc383cce700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc37c0f6510 0x7fc37c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:56:02.019 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:02.018+0000 7fc383cce700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc37c106c20 con 0x7fc37c0f6510 2024-01-18T06:56:02.019 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:02.018+0000 7fc381a6a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc37c0f6510 0x7fc37c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:56:02.019 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:02.018+0000 7fc381a6a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc37c0f6510 0x7fc37c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:59520/0 (socket says 172.21.15.194:59520) 2024-01-18T06:56:02.019 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:02.018+0000 7fc381a6a700 1 -- 172.21.15.194:0/1198669803 learned_addr learned my addr 172.21.15.194:0/1198669803 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:56:02.019 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:02.019+0000 7fc381a6a700 1 -- 172.21.15.194:0/1198669803 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc36c005040 con 0x7fc37c0f6510 2024-01-18T06:56:02.019 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:02.019+0000 7fc381a6a700 1 --2- 172.21.15.194:0/1198669803 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc37c0f6510 0x7fc37c106730 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7fc36c013510 tx=0x7fc36c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:56:02.019 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:02.019+0000 7fc3727fc700 1 -- 172.21.15.194:0/1198669803 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc36c009320 con 0x7fc37c0f6510 2024-01-18T06:56:02.019 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:02.019+0000 7fc383cce700 1 -- 172.21.15.194:0/1198669803 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc37c106e20 con 0x7fc37c0f6510 2024-01-18T06:56:02.019 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:02.019+0000 7fc3727fc700 1 -- 172.21.15.194:0/1198669803 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc36c013f00 con 0x7fc37c0f6510 2024-01-18T06:56:02.019 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:02.019+0000 7fc3727fc700 1 -- 172.21.15.194:0/1198669803 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc36c014d90 con 0x7fc37c0f6510 2024-01-18T06:56:02.019 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:02.019+0000 7fc383cce700 1 -- 172.21.15.194:0/1198669803 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc37c1072b0 con 0x7fc37c0f6510 2024-01-18T06:56:02.020 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:02.020+0000 7fc383cce700 1 -- 172.21.15.194:0/1198669803 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc37c045ad0 con 0x7fc37c0f6510 2024-01-18T06:56:02.020 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:02.020+0000 7fc3727fc700 1 -- 172.21.15.194:0/1198669803 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc36c0095f0 con 0x7fc37c0f6510 2024-01-18T06:56:02.020 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:02.020+0000 7fc3727fc700 1 --2- 172.21.15.194:0/1198669803 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc36802ff60 0x7fc368032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:56:02.020 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:02.020+0000 7fc3727fc700 1 -- 172.21.15.194:0/1198669803 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc36c040fc0 con 0x7fc37c0f6510 2024-01-18T06:56:02.020 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:02.020+0000 7fc381269700 1 --2- 172.21.15.194:0/1198669803 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc36802ff60 0x7fc368032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:56:02.021 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:02.021+0000 7fc381269700 1 --2- 172.21.15.194:0/1198669803 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc36802ff60 0x7fc368032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fc3780096a0 tx=0x7fc378006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:56:02.024 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:02.025+0000 7fc3727fc700 1 -- 172.21.15.194:0/1198669803 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc36c014420 con 0x7fc37c0f6510 2024-01-18T06:56:02.231 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:02.232+0000 7fc383cce700 1 -- 172.21.15.194:0/1198669803 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc37c0fbf60 con 0x7fc37c0f6510 2024-01-18T06:56:02.231 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:02.232+0000 7fc3727fc700 1 -- 172.21.15.194:0/1198669803 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc36c010230 con 0x7fc37c0f6510 2024-01-18T06:56:02.232 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:56:02.232 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:56:02.234 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:02.234+0000 7fc383cce700 1 -- 172.21.15.194:0/1198669803 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc36802ff60 msgr2=0x7fc368032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:02.234 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:02.234+0000 7fc383cce700 1 --2- 172.21.15.194:0/1198669803 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc36802ff60 0x7fc368032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fc3780096a0 tx=0x7fc378006b40).stop 2024-01-18T06:56:02.234 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:02.234+0000 7fc383cce700 1 -- 172.21.15.194:0/1198669803 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc37c0f6510 msgr2=0x7fc37c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:02.234 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:02.234+0000 7fc383cce700 1 --2- 172.21.15.194:0/1198669803 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc37c0f6510 0x7fc37c106730 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7fc36c013510 tx=0x7fc36c00b040).stop 2024-01-18T06:56:02.234 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:02.234+0000 7fc383cce700 1 -- 172.21.15.194:0/1198669803 shutdown_connections 2024-01-18T06:56:02.234 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:02.234+0000 7fc383cce700 1 --2- 172.21.15.194:0/1198669803 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc36802ff60 0x7fc368032420 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:02.234 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:02.234+0000 7fc383cce700 1 --2- 172.21.15.194:0/1198669803 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc37c0f6510 0x7fc37c106730 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:02.234 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:02.234+0000 7fc383cce700 1 -- 172.21.15.194:0/1198669803 >> 172.21.15.194:0/1198669803 conn(0x7fc37c0f1c70 msgr2=0x7fc37c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:56:02.234 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:02.234+0000 7fc383cce700 1 -- 172.21.15.194:0/1198669803 shutdown_connections 2024-01-18T06:56:02.234 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:02.234+0000 7fc383cce700 1 -- 172.21.15.194:0/1198669803 wait complete. 2024-01-18T06:56:02.235 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:56:03.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:03 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:56:02.046345+0000 mgr.smithi047.fuhzzr (mgr.14162) 234 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:56:03.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:03 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:56:02.233335+0000 mon.smithi047 (mon.0) 290 : audit [DBG] from='client.? 172.21.15.194:0/1198669803' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:56:03.607 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:56:03.607 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:56:04.763 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:04.763+0000 7faa8a355700 1 -- 172.21.15.194:0/2722397942 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faa840601e0 msgr2=0x7faa840f9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:04.763 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:04.763+0000 7faa8a355700 1 --2- 172.21.15.194:0/2722397942 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faa840601e0 0x7faa840f9170 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7faa74009fb0 tx=0x7faa74009b30).stop 2024-01-18T06:56:04.763 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:04.763+0000 7faa8a355700 1 -- 172.21.15.194:0/2722397942 shutdown_connections 2024-01-18T06:56:04.763 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:04.763+0000 7faa8a355700 1 --2- 172.21.15.194:0/2722397942 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faa840601e0 0x7faa840f9170 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:04.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:04.763+0000 7faa8a355700 1 -- 172.21.15.194:0/2722397942 >> 172.21.15.194:0/2722397942 conn(0x7faa840f0b70 msgr2=0x7faa840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:56:04.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:04.763+0000 7faa8a355700 1 -- 172.21.15.194:0/2722397942 shutdown_connections 2024-01-18T06:56:04.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:04.763+0000 7faa8a355700 1 -- 172.21.15.194:0/2722397942 wait complete. 2024-01-18T06:56:04.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:04.764+0000 7faa8a355700 1 Processor -- start 2024-01-18T06:56:04.766 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:04.764+0000 7faa8a355700 1 -- start start 2024-01-18T06:56:04.766 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:04.764+0000 7faa8a355700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faa840601e0 0x7faa840f6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:56:04.766 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:04.764+0000 7faa8a355700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa840f5070 con 0x7faa840601e0 2024-01-18T06:56:04.766 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:04.765+0000 7faa83fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faa840601e0 0x7faa840f6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:56:04.766 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:04.765+0000 7faa83fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faa840601e0 0x7faa840f6bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:59536/0 (socket says 172.21.15.194:59536) 2024-01-18T06:56:04.766 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:04.765+0000 7faa83fff700 1 -- 172.21.15.194:0/3332791885 learned_addr learned my addr 172.21.15.194:0/3332791885 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:56:04.766 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:04.765+0000 7faa83fff700 1 -- 172.21.15.194:0/3332791885 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa74005040 con 0x7faa840601e0 2024-01-18T06:56:04.766 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:04.765+0000 7faa83fff700 1 --2- 172.21.15.194:0/3332791885 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faa840601e0 0x7faa840f6bf0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7faa74004dd0 tx=0x7faa74009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:56:04.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:04.765+0000 7faa80ff9700 1 -- 172.21.15.194:0/3332791885 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa74018070 con 0x7faa840601e0 2024-01-18T06:56:04.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:04.765+0000 7faa8a355700 1 -- 172.21.15.194:0/3332791885 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa840f5210 con 0x7faa840601e0 2024-01-18T06:56:04.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:04.765+0000 7faa80ff9700 1 -- 172.21.15.194:0/3332791885 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faa74013410 con 0x7faa840601e0 2024-01-18T06:56:04.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:04.765+0000 7faa80ff9700 1 -- 172.21.15.194:0/3332791885 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa7401c3c0 con 0x7faa840601e0 2024-01-18T06:56:04.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:04.765+0000 7faa8a355700 1 -- 172.21.15.194:0/3332791885 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa840f56a0 con 0x7faa840601e0 2024-01-18T06:56:04.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:04.766+0000 7faa80ff9700 1 -- 172.21.15.194:0/3332791885 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faa7401c630 con 0x7faa840601e0 2024-01-18T06:56:04.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:04.766+0000 7faa80ff9700 1 --2- 172.21.15.194:0/3332791885 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7faa7002ff60 0x7faa70032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:56:04.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:04.766+0000 7faa80ff9700 1 -- 172.21.15.194:0/3332791885 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faa740414a0 con 0x7faa840601e0 2024-01-18T06:56:04.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:04.766+0000 7faa8a355700 1 -- 172.21.15.194:0/3332791885 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa84059100 con 0x7faa840601e0 2024-01-18T06:56:04.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:04.766+0000 7faa837fe700 1 --2- 172.21.15.194:0/3332791885 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7faa7002ff60 0x7faa70032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:56:04.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:04.767+0000 7faa837fe700 1 --2- 172.21.15.194:0/3332791885 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7faa7002ff60 0x7faa70032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7faa6c0096a0 tx=0x7faa6c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:56:04.770 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:04.771+0000 7faa80ff9700 1 -- 172.21.15.194:0/3332791885 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faa7401ca30 con 0x7faa840601e0 2024-01-18T06:56:04.980 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:04.980+0000 7faa8a355700 1 -- 172.21.15.194:0/3332791885 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faa840f5da0 con 0x7faa840601e0 2024-01-18T06:56:04.980 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:04.981+0000 7faa80ff9700 1 -- 172.21.15.194:0/3332791885 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7faa7401b9c0 con 0x7faa840601e0 2024-01-18T06:56:04.981 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:56:04.981 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:56:04.982 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:04.983+0000 7faa8a355700 1 -- 172.21.15.194:0/3332791885 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7faa7002ff60 msgr2=0x7faa70032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:04.983 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:04.983+0000 7faa8a355700 1 --2- 172.21.15.194:0/3332791885 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7faa7002ff60 0x7faa70032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7faa6c0096a0 tx=0x7faa6c006b40).stop 2024-01-18T06:56:04.983 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:04.983+0000 7faa8a355700 1 -- 172.21.15.194:0/3332791885 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faa840601e0 msgr2=0x7faa840f6bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:04.983 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:04.983+0000 7faa8a355700 1 --2- 172.21.15.194:0/3332791885 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faa840601e0 0x7faa840f6bf0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7faa74004dd0 tx=0x7faa74009b30).stop 2024-01-18T06:56:04.983 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:04.983+0000 7faa8a355700 1 -- 172.21.15.194:0/3332791885 shutdown_connections 2024-01-18T06:56:04.983 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:04.983+0000 7faa8a355700 1 --2- 172.21.15.194:0/3332791885 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7faa7002ff60 0x7faa70032420 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:04.983 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:04.983+0000 7faa8a355700 1 --2- 172.21.15.194:0/3332791885 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faa840601e0 0x7faa840f6bf0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:04.983 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:04.983+0000 7faa8a355700 1 -- 172.21.15.194:0/3332791885 >> 172.21.15.194:0/3332791885 conn(0x7faa840f0b70 msgr2=0x7faa840f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:56:04.983 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:04.983+0000 7faa8a355700 1 -- 172.21.15.194:0/3332791885 shutdown_connections 2024-01-18T06:56:04.983 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:04.983+0000 7faa8a355700 1 -- 172.21.15.194:0/3332791885 wait complete. 2024-01-18T06:56:04.983 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:56:05.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:05 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:56:04.046711+0000 mgr.smithi047.fuhzzr (mgr.14162) 235 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:56:05.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:05 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:56:04.981955+0000 mon.smithi047 (mon.0) 291 : audit [DBG] from='client.? 172.21.15.194:0/3332791885' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:56:06.355 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:56:06.355 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:56:07.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:07 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:56:06.047097+0000 mgr.smithi047.fuhzzr (mgr.14162) 236 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:56:07.557 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:07.557+0000 7f931359e700 1 -- 172.21.15.194:0/2188799503 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f93140f51a0 msgr2=0x7f93140f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:07.557 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:07.557+0000 7f931359e700 1 --2- 172.21.15.194:0/2188799503 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f93140f51a0 0x7f93140f55c0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f9308009fb0 tx=0x7f9308009b30).stop 2024-01-18T06:56:07.557 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:07.557+0000 7f931359e700 1 -- 172.21.15.194:0/2188799503 shutdown_connections 2024-01-18T06:56:07.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:07.557+0000 7f931359e700 1 --2- 172.21.15.194:0/2188799503 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f93140f51a0 0x7f93140f55c0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:07.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:07.557+0000 7f931359e700 1 -- 172.21.15.194:0/2188799503 >> 172.21.15.194:0/2188799503 conn(0x7f93140f09a0 msgr2=0x7f93140f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:56:07.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:07.557+0000 7f931359e700 1 -- 172.21.15.194:0/2188799503 shutdown_connections 2024-01-18T06:56:07.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:07.557+0000 7f931359e700 1 -- 172.21.15.194:0/2188799503 wait complete. 2024-01-18T06:56:07.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:07.557+0000 7f931359e700 1 Processor -- start 2024-01-18T06:56:07.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:07.557+0000 7f931359e700 1 -- start start 2024-01-18T06:56:07.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:07.557+0000 7f931359e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f93140f51a0 0x7f93141053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:56:07.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:07.557+0000 7f931359e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93141058a0 con 0x7f93140f51a0 2024-01-18T06:56:07.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:07.558+0000 7f931259c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f93140f51a0 0x7f93141053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:56:07.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:07.558+0000 7f931259c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f93140f51a0 0x7f93141053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:36252/0 (socket says 172.21.15.194:36252) 2024-01-18T06:56:07.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:07.558+0000 7f931259c700 1 -- 172.21.15.194:0/3721735923 learned_addr learned my addr 172.21.15.194:0/3721735923 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:56:07.560 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:07.558+0000 7f931259c700 1 -- 172.21.15.194:0/3721735923 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9308005040 con 0x7f93140f51a0 2024-01-18T06:56:07.560 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:07.558+0000 7f931259c700 1 --2- 172.21.15.194:0/3721735923 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f93140f51a0 0x7f93141053b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f9308013510 tx=0x7f930800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:56:07.560 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:07.559+0000 7f9302ffd700 1 -- 172.21.15.194:0/3721735923 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9308009320 con 0x7f93140f51a0 2024-01-18T06:56:07.560 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:07.559+0000 7f931359e700 1 -- 172.21.15.194:0/3721735923 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9314105aa0 con 0x7f93140f51a0 2024-01-18T06:56:07.560 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:07.559+0000 7f9302ffd700 1 -- 172.21.15.194:0/3721735923 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9308013f00 con 0x7f93140f51a0 2024-01-18T06:56:07.560 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:07.559+0000 7f9302ffd700 1 -- 172.21.15.194:0/3721735923 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9308014dc0 con 0x7f93140f51a0 2024-01-18T06:56:07.560 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:07.559+0000 7f931359e700 1 -- 172.21.15.194:0/3721735923 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9314105f30 con 0x7f93140f51a0 2024-01-18T06:56:07.561 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:07.559+0000 7f9302ffd700 1 -- 172.21.15.194:0/3721735923 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9308014420 con 0x7f93140f51a0 2024-01-18T06:56:07.561 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:07.560+0000 7f9302ffd700 1 --2- 172.21.15.194:0/3721735923 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f92fc02ff60 0x7f92fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:56:07.561 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:07.560+0000 7f9302ffd700 1 -- 172.21.15.194:0/3721735923 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9308042e90 con 0x7f93140f51a0 2024-01-18T06:56:07.561 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:07.560+0000 7f931359e700 1 -- 172.21.15.194:0/3721735923 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9314045ad0 con 0x7f93140f51a0 2024-01-18T06:56:07.561 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:07.560+0000 7f9311d9b700 1 --2- 172.21.15.194:0/3721735923 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f92fc02ff60 0x7f92fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:56:07.561 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:07.560+0000 7f9311d9b700 1 --2- 172.21.15.194:0/3721735923 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f92fc02ff60 0x7f92fc032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f93040096a0 tx=0x7f9304006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:56:07.564 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:07.565+0000 7f9302ffd700 1 -- 172.21.15.194:0/3721735923 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f930800fcb0 con 0x7f93140f51a0 2024-01-18T06:56:07.770 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:07.770+0000 7f931359e700 1 -- 172.21.15.194:0/3721735923 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9314106de0 con 0x7f93140f51a0 2024-01-18T06:56:07.770 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:07.771+0000 7f9302ffd700 1 -- 172.21.15.194:0/3721735923 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f930801fb70 con 0x7f93140f51a0 2024-01-18T06:56:07.771 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:56:07.771 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:56:07.772 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:07.773+0000 7f931359e700 1 -- 172.21.15.194:0/3721735923 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f92fc02ff60 msgr2=0x7f92fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:07.773 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:07.773+0000 7f931359e700 1 --2- 172.21.15.194:0/3721735923 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f92fc02ff60 0x7f92fc032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f93040096a0 tx=0x7f9304006b40).stop 2024-01-18T06:56:07.773 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:07.773+0000 7f931359e700 1 -- 172.21.15.194:0/3721735923 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f93140f51a0 msgr2=0x7f93141053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:07.773 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:07.773+0000 7f931359e700 1 --2- 172.21.15.194:0/3721735923 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f93140f51a0 0x7f93141053b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f9308013510 tx=0x7f930800b040).stop 2024-01-18T06:56:07.773 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:07.773+0000 7f931359e700 1 -- 172.21.15.194:0/3721735923 shutdown_connections 2024-01-18T06:56:07.773 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:07.773+0000 7f931359e700 1 --2- 172.21.15.194:0/3721735923 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f92fc02ff60 0x7f92fc032420 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:07.773 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:07.773+0000 7f931359e700 1 --2- 172.21.15.194:0/3721735923 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f93140f51a0 0x7f93141053b0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:07.773 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:07.773+0000 7f931359e700 1 -- 172.21.15.194:0/3721735923 >> 172.21.15.194:0/3721735923 conn(0x7f93140f09a0 msgr2=0x7f93140f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:56:07.773 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:07.773+0000 7f931359e700 1 -- 172.21.15.194:0/3721735923 shutdown_connections 2024-01-18T06:56:07.773 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:07.773+0000 7f931359e700 1 -- 172.21.15.194:0/3721735923 wait complete. 2024-01-18T06:56:07.773 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:56:08.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:08 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:56:07.771997+0000 mon.smithi047 (mon.0) 292 : audit [DBG] from='client.? 172.21.15.194:0/3721735923' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:56:09.156 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:56:09.156 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:56:09.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:09 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:56:08.047369+0000 mgr.smithi047.fuhzzr (mgr.14162) 237 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:56:10.350 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:10.349+0000 7f4d6e21a700 1 -- 172.21.15.194:0/3961532059 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4d680f62a0 msgr2=0x7f4d680f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:10.350 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:10.349+0000 7f4d6e21a700 1 --2- 172.21.15.194:0/3961532059 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4d680f62a0 0x7f4d680f66c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f4d58009fb0 tx=0x7f4d58009b30).stop 2024-01-18T06:56:10.350 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:10.349+0000 7f4d6e21a700 1 -- 172.21.15.194:0/3961532059 shutdown_connections 2024-01-18T06:56:10.350 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:10.349+0000 7f4d6e21a700 1 --2- 172.21.15.194:0/3961532059 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4d680f62a0 0x7f4d680f66c0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:10.350 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:10.349+0000 7f4d6e21a700 1 -- 172.21.15.194:0/3961532059 >> 172.21.15.194:0/3961532059 conn(0x7f4d680f1aa0 msgr2=0x7f4d680f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:56:10.350 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:10.349+0000 7f4d6e21a700 1 -- 172.21.15.194:0/3961532059 shutdown_connections 2024-01-18T06:56:10.350 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:10.349+0000 7f4d6e21a700 1 -- 172.21.15.194:0/3961532059 wait complete. 2024-01-18T06:56:10.350 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:10.349+0000 7f4d6e21a700 1 Processor -- start 2024-01-18T06:56:10.350 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:10.349+0000 7f4d6e21a700 1 -- start start 2024-01-18T06:56:10.350 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:10.349+0000 7f4d6e21a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4d680f62a0 0x7f4d681064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:56:10.351 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:10.349+0000 7f4d6e21a700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d681069a0 con 0x7f4d680f62a0 2024-01-18T06:56:10.351 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:10.350+0000 7f4d6d218700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4d680f62a0 0x7f4d681064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:56:10.351 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:10.350+0000 7f4d6d218700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4d680f62a0 0x7f4d681064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:36262/0 (socket says 172.21.15.194:36262) 2024-01-18T06:56:10.351 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:10.350+0000 7f4d6d218700 1 -- 172.21.15.194:0/3349714608 learned_addr learned my addr 172.21.15.194:0/3349714608 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:56:10.351 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:10.350+0000 7f4d6d218700 1 -- 172.21.15.194:0/3349714608 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d58005040 con 0x7f4d680f62a0 2024-01-18T06:56:10.351 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:10.350+0000 7f4d6d218700 1 --2- 172.21.15.194:0/3349714608 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4d680f62a0 0x7f4d681064b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f4d58013510 tx=0x7f4d5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:56:10.351 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:10.350+0000 7f4d5dffb700 1 -- 172.21.15.194:0/3349714608 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d58009320 con 0x7f4d680f62a0 2024-01-18T06:56:10.351 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:10.350+0000 7f4d6e21a700 1 -- 172.21.15.194:0/3349714608 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d68106ba0 con 0x7f4d680f62a0 2024-01-18T06:56:10.351 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:10.350+0000 7f4d5dffb700 1 -- 172.21.15.194:0/3349714608 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4d58013f00 con 0x7f4d680f62a0 2024-01-18T06:56:10.351 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:10.350+0000 7f4d5dffb700 1 -- 172.21.15.194:0/3349714608 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d58014dc0 con 0x7f4d680f62a0 2024-01-18T06:56:10.351 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:10.350+0000 7f4d6e21a700 1 -- 172.21.15.194:0/3349714608 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d68107030 con 0x7f4d680f62a0 2024-01-18T06:56:10.353 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:10.351+0000 7f4d5dffb700 1 -- 172.21.15.194:0/3349714608 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4d580095f0 con 0x7f4d680f62a0 2024-01-18T06:56:10.353 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:10.351+0000 7f4d6e21a700 1 -- 172.21.15.194:0/3349714608 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d68045ad0 con 0x7f4d680f62a0 2024-01-18T06:56:10.353 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:10.351+0000 7f4d5dffb700 1 --2- 172.21.15.194:0/3349714608 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f4d5402ff60 0x7f4d54032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:56:10.353 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:10.351+0000 7f4d5dffb700 1 -- 172.21.15.194:0/3349714608 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4d58041040 con 0x7f4d680f62a0 2024-01-18T06:56:10.353 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:10.352+0000 7f4d6ca17700 1 --2- 172.21.15.194:0/3349714608 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f4d5402ff60 0x7f4d54032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:56:10.353 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:10.352+0000 7f4d6ca17700 1 --2- 172.21.15.194:0/3349714608 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f4d5402ff60 0x7f4d54032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f4d640096a0 tx=0x7f4d64006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:56:10.355 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:10.356+0000 7f4d5dffb700 1 -- 172.21.15.194:0/3349714608 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4d58014420 con 0x7f4d680f62a0 2024-01-18T06:56:10.564 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:10.565+0000 7f4d6e21a700 1 -- 172.21.15.194:0/3349714608 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4d68107e70 con 0x7f4d680f62a0 2024-01-18T06:56:10.565 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:10.565+0000 7f4d5dffb700 1 -- 172.21.15.194:0/3349714608 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4d58010230 con 0x7f4d680f62a0 2024-01-18T06:56:10.565 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:56:10.565 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:56:10.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:10.567+0000 7f4d6e21a700 1 -- 172.21.15.194:0/3349714608 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f4d5402ff60 msgr2=0x7f4d54032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:10.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:10.567+0000 7f4d6e21a700 1 --2- 172.21.15.194:0/3349714608 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f4d5402ff60 0x7f4d54032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f4d640096a0 tx=0x7f4d64006b40).stop 2024-01-18T06:56:10.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:10.567+0000 7f4d6e21a700 1 -- 172.21.15.194:0/3349714608 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4d680f62a0 msgr2=0x7f4d681064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:10.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:10.567+0000 7f4d6e21a700 1 --2- 172.21.15.194:0/3349714608 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4d680f62a0 0x7f4d681064b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f4d58013510 tx=0x7f4d5800b040).stop 2024-01-18T06:56:10.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:10.567+0000 7f4d6e21a700 1 -- 172.21.15.194:0/3349714608 shutdown_connections 2024-01-18T06:56:10.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:10.567+0000 7f4d6e21a700 1 --2- 172.21.15.194:0/3349714608 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f4d5402ff60 0x7f4d54032420 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:10.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:10.567+0000 7f4d6e21a700 1 --2- 172.21.15.194:0/3349714608 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4d680f62a0 0x7f4d681064b0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:10.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:10.567+0000 7f4d6e21a700 1 -- 172.21.15.194:0/3349714608 >> 172.21.15.194:0/3349714608 conn(0x7f4d680f1aa0 msgr2=0x7f4d680f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:56:10.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:10.567+0000 7f4d6e21a700 1 -- 172.21.15.194:0/3349714608 shutdown_connections 2024-01-18T06:56:10.568 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:10.567+0000 7f4d6e21a700 1 -- 172.21.15.194:0/3349714608 wait complete. 2024-01-18T06:56:10.568 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:56:11.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:11 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:56:10.047722+0000 mgr.smithi047.fuhzzr (mgr.14162) 238 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:56:11.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:11 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:56:10.566542+0000 mon.smithi047 (mon.0) 293 : audit [DBG] from='client.? 172.21.15.194:0/3349714608' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:56:11.946 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:56:11.947 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:56:13.076 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:13.076+0000 7fb3a96c2700 1 -- 172.21.15.194:0/3303253997 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3a40601e0 msgr2=0x7fb3a40f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:13.077 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:13.076+0000 7fb3a96c2700 1 --2- 172.21.15.194:0/3303253997 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3a40601e0 0x7fb3a40f8f20 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7fb394009fb0 tx=0x7fb394009b30).stop 2024-01-18T06:56:13.077 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:13.076+0000 7fb3a96c2700 1 -- 172.21.15.194:0/3303253997 shutdown_connections 2024-01-18T06:56:13.077 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:13.076+0000 7fb3a96c2700 1 --2- 172.21.15.194:0/3303253997 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3a40601e0 0x7fb3a40f8f20 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:13.077 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:13.076+0000 7fb3a96c2700 1 -- 172.21.15.194:0/3303253997 >> 172.21.15.194:0/3303253997 conn(0x7fb3a40f09a0 msgr2=0x7fb3a40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:56:13.077 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:13.076+0000 7fb3a96c2700 1 -- 172.21.15.194:0/3303253997 shutdown_connections 2024-01-18T06:56:13.077 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:13.076+0000 7fb3a96c2700 1 -- 172.21.15.194:0/3303253997 wait complete. 2024-01-18T06:56:13.077 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:13.076+0000 7fb3a96c2700 1 Processor -- start 2024-01-18T06:56:13.078 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:13.076+0000 7fb3a96c2700 1 -- start start 2024-01-18T06:56:13.078 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:13.077+0000 7fb3a96c2700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3a40601e0 0x7fb3a40f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:56:13.078 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:13.077+0000 7fb3a96c2700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a40f4e20 con 0x7fb3a40601e0 2024-01-18T06:56:13.079 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:13.077+0000 7fb3a3fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3a40601e0 0x7fb3a40f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:56:13.079 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:13.077+0000 7fb3a3fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3a40601e0 0x7fb3a40f69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:36278/0 (socket says 172.21.15.194:36278) 2024-01-18T06:56:13.079 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:13.077+0000 7fb3a3fff700 1 -- 172.21.15.194:0/3544150637 learned_addr learned my addr 172.21.15.194:0/3544150637 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:56:13.079 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:13.077+0000 7fb3a3fff700 1 -- 172.21.15.194:0/3544150637 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb394005040 con 0x7fb3a40601e0 2024-01-18T06:56:13.079 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:13.077+0000 7fb3a3fff700 1 --2- 172.21.15.194:0/3544150637 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3a40601e0 0x7fb3a40f69a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fb394014000 tx=0x7fb39400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:56:13.079 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:13.078+0000 7fb3a0ff9700 1 -- 172.21.15.194:0/3544150637 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3940095f0 con 0x7fb3a40601e0 2024-01-18T06:56:13.079 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:13.078+0000 7fb3a96c2700 1 -- 172.21.15.194:0/3544150637 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3a40f4fc0 con 0x7fb3a40601e0 2024-01-18T06:56:13.079 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:13.078+0000 7fb3a0ff9700 1 -- 172.21.15.194:0/3544150637 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb394007e70 con 0x7fb3a40601e0 2024-01-18T06:56:13.079 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:13.078+0000 7fb3a0ff9700 1 -- 172.21.15.194:0/3544150637 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb394014ea0 con 0x7fb3a40601e0 2024-01-18T06:56:13.079 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:13.078+0000 7fb3a96c2700 1 -- 172.21.15.194:0/3544150637 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3a40f5450 con 0x7fb3a40601e0 2024-01-18T06:56:13.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:13.078+0000 7fb3a96c2700 1 -- 172.21.15.194:0/3544150637 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3a4059100 con 0x7fb3a40601e0 2024-01-18T06:56:13.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:13.079+0000 7fb3a0ff9700 1 -- 172.21.15.194:0/3544150637 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb394025070 con 0x7fb3a40601e0 2024-01-18T06:56:13.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:13.079+0000 7fb3a0ff9700 1 --2- 172.21.15.194:0/3544150637 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb38c02ff60 0x7fb38c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:56:13.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:13.079+0000 7fb3a0ff9700 1 -- 172.21.15.194:0/3544150637 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb3940409b0 con 0x7fb3a40601e0 2024-01-18T06:56:13.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:13.079+0000 7fb3a37fe700 1 --2- 172.21.15.194:0/3544150637 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb38c02ff60 0x7fb38c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:56:13.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:13.079+0000 7fb3a37fe700 1 --2- 172.21.15.194:0/3544150637 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb38c02ff60 0x7fb38c032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fb3980096a0 tx=0x7fb398006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:56:13.084 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:13.084+0000 7fb3a0ff9700 1 -- 172.21.15.194:0/3544150637 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb3940079c0 con 0x7fb3a40601e0 2024-01-18T06:56:13.291 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:13.290+0000 7fb3a96c2700 1 -- 172.21.15.194:0/3544150637 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb3a4045ad0 con 0x7fb3a40601e0 2024-01-18T06:56:13.291 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:13.291+0000 7fb3a0ff9700 1 -- 172.21.15.194:0/3544150637 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb39400f300 con 0x7fb3a40601e0 2024-01-18T06:56:13.292 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:56:13.292 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:56:13.293 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:13.293+0000 7fb3a96c2700 1 -- 172.21.15.194:0/3544150637 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb38c02ff60 msgr2=0x7fb38c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:13.293 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:13.293+0000 7fb3a96c2700 1 --2- 172.21.15.194:0/3544150637 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb38c02ff60 0x7fb38c032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fb3980096a0 tx=0x7fb398006b40).stop 2024-01-18T06:56:13.293 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:13.293+0000 7fb3a96c2700 1 -- 172.21.15.194:0/3544150637 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3a40601e0 msgr2=0x7fb3a40f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:13.293 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:13.293+0000 7fb3a96c2700 1 --2- 172.21.15.194:0/3544150637 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3a40601e0 0x7fb3a40f69a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fb394014000 tx=0x7fb39400b040).stop 2024-01-18T06:56:13.293 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:13.293+0000 7fb3a96c2700 1 -- 172.21.15.194:0/3544150637 shutdown_connections 2024-01-18T06:56:13.293 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:13.293+0000 7fb3a96c2700 1 --2- 172.21.15.194:0/3544150637 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb38c02ff60 0x7fb38c032420 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:13.293 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:13.293+0000 7fb3a96c2700 1 --2- 172.21.15.194:0/3544150637 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3a40601e0 0x7fb3a40f69a0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:13.293 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:13.293+0000 7fb3a96c2700 1 -- 172.21.15.194:0/3544150637 >> 172.21.15.194:0/3544150637 conn(0x7fb3a40f09a0 msgr2=0x7fb3a40f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:56:13.294 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:13.293+0000 7fb3a96c2700 1 -- 172.21.15.194:0/3544150637 shutdown_connections 2024-01-18T06:56:13.294 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:13.293+0000 7fb3a96c2700 1 -- 172.21.15.194:0/3544150637 wait complete. 2024-01-18T06:56:13.294 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:56:13.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:13 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:56:12.048113+0000 mgr.smithi047.fuhzzr (mgr.14162) 239 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:56:14.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:14 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:56:13.293042+0000 mon.smithi047 (mon.0) 294 : audit [DBG] from='client.? 172.21.15.194:0/3544150637' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:56:14.675 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:56:14.676 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:56:15.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:15 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:56:14.048533+0000 mgr.smithi047.fuhzzr (mgr.14162) 240 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:56:15.845 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:15.844+0000 7f6e93f96700 1 -- 172.21.15.194:0/1866950393 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e8c0f5410 msgr2=0x7f6e8c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:15.845 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:15.844+0000 7f6e93f96700 1 --2- 172.21.15.194:0/1866950393 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e8c0f5410 0x7f6e8c0f5830 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f6e88009fb0 tx=0x7f6e88009b30).stop 2024-01-18T06:56:15.846 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:15.844+0000 7f6e93f96700 1 -- 172.21.15.194:0/1866950393 shutdown_connections 2024-01-18T06:56:15.846 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:15.844+0000 7f6e93f96700 1 --2- 172.21.15.194:0/1866950393 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e8c0f5410 0x7f6e8c0f5830 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:15.846 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:15.844+0000 7f6e93f96700 1 -- 172.21.15.194:0/1866950393 >> 172.21.15.194:0/1866950393 conn(0x7f6e8c0f0b70 msgr2=0x7f6e8c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:56:15.846 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:15.844+0000 7f6e93f96700 1 -- 172.21.15.194:0/1866950393 shutdown_connections 2024-01-18T06:56:15.846 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:15.844+0000 7f6e93f96700 1 -- 172.21.15.194:0/1866950393 wait complete. 2024-01-18T06:56:15.846 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:15.845+0000 7f6e93f96700 1 Processor -- start 2024-01-18T06:56:15.846 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:15.845+0000 7f6e93f96700 1 -- start start 2024-01-18T06:56:15.846 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:15.845+0000 7f6e93f96700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e8c0f5410 0x7f6e8c109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:56:15.846 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:15.845+0000 7f6e93f96700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8c060240 con 0x7f6e8c0f5410 2024-01-18T06:56:15.848 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:15.845+0000 7f6e91d32700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e8c0f5410 0x7f6e8c109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:56:15.848 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:15.845+0000 7f6e91d32700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e8c0f5410 0x7f6e8c109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:36298/0 (socket says 172.21.15.194:36298) 2024-01-18T06:56:15.848 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:15.845+0000 7f6e91d32700 1 -- 172.21.15.194:0/2011121134 learned_addr learned my addr 172.21.15.194:0/2011121134 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:56:15.848 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:15.846+0000 7f6e91d32700 1 -- 172.21.15.194:0/2011121134 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e88005040 con 0x7f6e8c0f5410 2024-01-18T06:56:15.848 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:15.846+0000 7f6e91d32700 1 --2- 172.21.15.194:0/2011121134 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e8c0f5410 0x7f6e8c109400 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f6e88013510 tx=0x7f6e8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:56:15.848 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:15.846+0000 7f6e827fc700 1 -- 172.21.15.194:0/2011121134 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e88009320 con 0x7f6e8c0f5410 2024-01-18T06:56:15.848 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:15.846+0000 7f6e93f96700 1 -- 172.21.15.194:0/2011121134 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e8c060440 con 0x7f6e8c0f5410 2024-01-18T06:56:15.848 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:15.846+0000 7f6e827fc700 1 -- 172.21.15.194:0/2011121134 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6e88013f00 con 0x7f6e8c0f5410 2024-01-18T06:56:15.848 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:15.846+0000 7f6e827fc700 1 -- 172.21.15.194:0/2011121134 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e88014d90 con 0x7f6e8c0f5410 2024-01-18T06:56:15.848 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:15.846+0000 7f6e93f96700 1 -- 172.21.15.194:0/2011121134 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e8c109ab0 con 0x7f6e8c0f5410 2024-01-18T06:56:15.848 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:15.847+0000 7f6e93f96700 1 -- 172.21.15.194:0/2011121134 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e8c045ad0 con 0x7f6e8c0f5410 2024-01-18T06:56:15.848 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:15.847+0000 7f6e827fc700 1 -- 172.21.15.194:0/2011121134 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6e880095f0 con 0x7f6e8c0f5410 2024-01-18T06:56:15.848 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:15.847+0000 7f6e827fc700 1 --2- 172.21.15.194:0/2011121134 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f6e7802ff60 0x7f6e78032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:56:15.849 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:15.847+0000 7f6e827fc700 1 -- 172.21.15.194:0/2011121134 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6e88041080 con 0x7f6e8c0f5410 2024-01-18T06:56:15.849 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:15.847+0000 7f6e91531700 1 --2- 172.21.15.194:0/2011121134 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f6e7802ff60 0x7f6e78032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:56:15.849 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:15.848+0000 7f6e91531700 1 --2- 172.21.15.194:0/2011121134 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f6e7802ff60 0x7f6e78032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f6e7c0096a0 tx=0x7f6e7c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:56:15.852 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:15.852+0000 7f6e827fc700 1 -- 172.21.15.194:0/2011121134 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6e88014420 con 0x7f6e8c0f5410 2024-01-18T06:56:16.059 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:16.058+0000 7f6e93f96700 1 -- 172.21.15.194:0/2011121134 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6e8c0fae60 con 0x7f6e8c0f5410 2024-01-18T06:56:16.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:16.059+0000 7f6e827fc700 1 -- 172.21.15.194:0/2011121134 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6e88010230 con 0x7f6e8c0f5410 2024-01-18T06:56:16.060 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:56:16.060 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:56:16.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:16.061+0000 7f6e93f96700 1 -- 172.21.15.194:0/2011121134 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f6e7802ff60 msgr2=0x7f6e78032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:16.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:16.061+0000 7f6e93f96700 1 --2- 172.21.15.194:0/2011121134 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f6e7802ff60 0x7f6e78032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f6e7c0096a0 tx=0x7f6e7c006b40).stop 2024-01-18T06:56:16.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:16.061+0000 7f6e93f96700 1 -- 172.21.15.194:0/2011121134 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e8c0f5410 msgr2=0x7f6e8c109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:16.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:16.061+0000 7f6e93f96700 1 --2- 172.21.15.194:0/2011121134 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e8c0f5410 0x7f6e8c109400 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f6e88013510 tx=0x7f6e8800b040).stop 2024-01-18T06:56:16.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:16.061+0000 7f6e93f96700 1 -- 172.21.15.194:0/2011121134 shutdown_connections 2024-01-18T06:56:16.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:16.061+0000 7f6e93f96700 1 --2- 172.21.15.194:0/2011121134 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f6e7802ff60 0x7f6e78032420 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:16.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:16.061+0000 7f6e93f96700 1 --2- 172.21.15.194:0/2011121134 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6e8c0f5410 0x7f6e8c109400 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:16.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:16.061+0000 7f6e93f96700 1 -- 172.21.15.194:0/2011121134 >> 172.21.15.194:0/2011121134 conn(0x7f6e8c0f0b70 msgr2=0x7f6e8c0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:56:16.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:16.061+0000 7f6e93f96700 1 -- 172.21.15.194:0/2011121134 shutdown_connections 2024-01-18T06:56:16.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:16.061+0000 7f6e93f96700 1 -- 172.21.15.194:0/2011121134 wait complete. 2024-01-18T06:56:16.063 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:56:16.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:16 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:56:16.061136+0000 mon.smithi047 (mon.0) 295 : audit [DBG] from='client.? 172.21.15.194:0/2011121134' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:56:17.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:17 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:56:16.048833+0000 mgr.smithi047.fuhzzr (mgr.14162) 241 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:56:17.468 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:56:17.469 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:56:18.623 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:18.621+0000 7f6f719a0700 1 -- 172.21.15.194:0/1308135855 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6f6c0601e0 msgr2=0x7f6f6c0f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:18.623 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:18.621+0000 7f6f719a0700 1 --2- 172.21.15.194:0/1308135855 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6f6c0601e0 0x7f6f6c0f8f20 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f6f60009fb0 tx=0x7f6f60009b30).stop 2024-01-18T06:56:18.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:18.621+0000 7f6f719a0700 1 -- 172.21.15.194:0/1308135855 shutdown_connections 2024-01-18T06:56:18.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:18.621+0000 7f6f719a0700 1 --2- 172.21.15.194:0/1308135855 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6f6c0601e0 0x7f6f6c0f8f20 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:18.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:18.621+0000 7f6f719a0700 1 -- 172.21.15.194:0/1308135855 >> 172.21.15.194:0/1308135855 conn(0x7f6f6c0f09a0 msgr2=0x7f6f6c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:56:18.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:18.621+0000 7f6f719a0700 1 -- 172.21.15.194:0/1308135855 shutdown_connections 2024-01-18T06:56:18.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:18.621+0000 7f6f719a0700 1 -- 172.21.15.194:0/1308135855 wait complete. 2024-01-18T06:56:18.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:18.622+0000 7f6f719a0700 1 Processor -- start 2024-01-18T06:56:18.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:18.622+0000 7f6f719a0700 1 -- start start 2024-01-18T06:56:18.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:18.622+0000 7f6f719a0700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6f6c0601e0 0x7f6f6c106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:56:18.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:18.622+0000 7f6f719a0700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6c1069f0 con 0x7f6f6c0601e0 2024-01-18T06:56:18.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:18.622+0000 7f6f7099e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6f6c0601e0 0x7f6f6c106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:56:18.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:18.622+0000 7f6f7099e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6f6c0601e0 0x7f6f6c106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:58000/0 (socket says 172.21.15.194:58000) 2024-01-18T06:56:18.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:18.622+0000 7f6f7099e700 1 -- 172.21.15.194:0/1366008014 learned_addr learned my addr 172.21.15.194:0/1366008014 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:56:18.625 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:18.623+0000 7f6f7099e700 1 -- 172.21.15.194:0/1366008014 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f60005040 con 0x7f6f6c0601e0 2024-01-18T06:56:18.625 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:18.623+0000 7f6f7099e700 1 --2- 172.21.15.194:0/1366008014 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6f6c0601e0 0x7f6f6c106500 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f6f60013510 tx=0x7f6f6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:56:18.625 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:18.623+0000 7f6f697fa700 1 -- 172.21.15.194:0/1366008014 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f60009320 con 0x7f6f6c0601e0 2024-01-18T06:56:18.625 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:18.623+0000 7f6f719a0700 1 -- 172.21.15.194:0/1366008014 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f6c106bf0 con 0x7f6f6c0601e0 2024-01-18T06:56:18.625 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:18.623+0000 7f6f697fa700 1 -- 172.21.15.194:0/1366008014 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6f60013f00 con 0x7f6f6c0601e0 2024-01-18T06:56:18.625 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:18.623+0000 7f6f697fa700 1 -- 172.21.15.194:0/1366008014 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f60014d90 con 0x7f6f6c0601e0 2024-01-18T06:56:18.625 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:18.623+0000 7f6f719a0700 1 -- 172.21.15.194:0/1366008014 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f6c107080 con 0x7f6f6c0601e0 2024-01-18T06:56:18.626 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:18.624+0000 7f6f697fa700 1 -- 172.21.15.194:0/1366008014 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6f600095f0 con 0x7f6f6c0601e0 2024-01-18T06:56:18.626 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:18.624+0000 7f6f719a0700 1 -- 172.21.15.194:0/1366008014 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f6c045ad0 con 0x7f6f6c0601e0 2024-01-18T06:56:18.626 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:18.624+0000 7f6f697fa700 1 --2- 172.21.15.194:0/1366008014 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f6f5c02ff60 0x7f6f5c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:56:18.626 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:18.624+0000 7f6f697fa700 1 -- 172.21.15.194:0/1366008014 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6f60041000 con 0x7f6f6c0601e0 2024-01-18T06:56:18.626 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:18.624+0000 7f6f6bfff700 1 --2- 172.21.15.194:0/1366008014 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f6f5c02ff60 0x7f6f5c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:56:18.626 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:18.625+0000 7f6f6bfff700 1 --2- 172.21.15.194:0/1366008014 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f6f5c02ff60 0x7f6f5c032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f6f580096a0 tx=0x7f6f58006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:56:18.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:18.630+0000 7f6f697fa700 1 -- 172.21.15.194:0/1366008014 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6f60014420 con 0x7f6f6c0601e0 2024-01-18T06:56:18.836 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:18.836+0000 7f6f719a0700 1 -- 172.21.15.194:0/1366008014 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6f6c107f30 con 0x7f6f6c0601e0 2024-01-18T06:56:18.837 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:18.836+0000 7f6f697fa700 1 -- 172.21.15.194:0/1366008014 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6f60010230 con 0x7f6f6c0601e0 2024-01-18T06:56:18.837 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:56:18.837 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:56:18.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:18.838+0000 7f6f719a0700 1 -- 172.21.15.194:0/1366008014 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f6f5c02ff60 msgr2=0x7f6f5c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:18.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:18.838+0000 7f6f719a0700 1 --2- 172.21.15.194:0/1366008014 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f6f5c02ff60 0x7f6f5c032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f6f580096a0 tx=0x7f6f58006b40).stop 2024-01-18T06:56:18.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:18.838+0000 7f6f719a0700 1 -- 172.21.15.194:0/1366008014 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6f6c0601e0 msgr2=0x7f6f6c106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:18.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:18.838+0000 7f6f719a0700 1 --2- 172.21.15.194:0/1366008014 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6f6c0601e0 0x7f6f6c106500 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f6f60013510 tx=0x7f6f6000b040).stop 2024-01-18T06:56:18.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:18.838+0000 7f6f719a0700 1 -- 172.21.15.194:0/1366008014 shutdown_connections 2024-01-18T06:56:18.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:18.838+0000 7f6f719a0700 1 --2- 172.21.15.194:0/1366008014 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f6f5c02ff60 0x7f6f5c032420 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:18.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:18.838+0000 7f6f719a0700 1 --2- 172.21.15.194:0/1366008014 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6f6c0601e0 0x7f6f6c106500 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:18.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:18.838+0000 7f6f719a0700 1 -- 172.21.15.194:0/1366008014 >> 172.21.15.194:0/1366008014 conn(0x7f6f6c0f09a0 msgr2=0x7f6f6c0f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:56:18.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:18.838+0000 7f6f719a0700 1 -- 172.21.15.194:0/1366008014 shutdown_connections 2024-01-18T06:56:18.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:18.838+0000 7f6f719a0700 1 -- 172.21.15.194:0/1366008014 wait complete. 2024-01-18T06:56:18.840 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:56:19.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:18 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:56:18.049248+0000 mgr.smithi047.fuhzzr (mgr.14162) 242 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:56:19.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:18 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:56:18.838477+0000 mon.smithi047 (mon.0) 296 : audit [DBG] from='client.? 172.21.15.194:0/1366008014' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:56:20.251 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:56:20.251 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:56:21.412 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:21.411+0000 7f92d1d04700 1 -- 172.21.15.194:0/3681436392 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f92cc0f5410 msgr2=0x7f92cc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:21.412 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:21.411+0000 7f92d1d04700 1 --2- 172.21.15.194:0/3681436392 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f92cc0f5410 0x7f92cc0f5830 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f92c0009fb0 tx=0x7f92c0009b30).stop 2024-01-18T06:56:21.412 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:21.411+0000 7f92d1d04700 1 -- 172.21.15.194:0/3681436392 shutdown_connections 2024-01-18T06:56:21.412 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:21.411+0000 7f92d1d04700 1 --2- 172.21.15.194:0/3681436392 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f92cc0f5410 0x7f92cc0f5830 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:21.412 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:21.411+0000 7f92d1d04700 1 -- 172.21.15.194:0/3681436392 >> 172.21.15.194:0/3681436392 conn(0x7f92cc0f0b70 msgr2=0x7f92cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:56:21.412 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:21.411+0000 7f92d1d04700 1 -- 172.21.15.194:0/3681436392 shutdown_connections 2024-01-18T06:56:21.412 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:21.411+0000 7f92d1d04700 1 -- 172.21.15.194:0/3681436392 wait complete. 2024-01-18T06:56:21.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:21.412+0000 7f92d1d04700 1 Processor -- start 2024-01-18T06:56:21.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:21.412+0000 7f92d1d04700 1 -- start start 2024-01-18T06:56:21.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:21.412+0000 7f92d1d04700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f92cc0f5410 0x7f92cc105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:56:21.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:21.412+0000 7f92d1d04700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92cc105b20 con 0x7f92cc0f5410 2024-01-18T06:56:21.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:21.413+0000 7f92cb7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f92cc0f5410 0x7f92cc105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:56:21.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:21.413+0000 7f92cb7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f92cc0f5410 0x7f92cc105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:58020/0 (socket says 172.21.15.194:58020) 2024-01-18T06:56:21.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:21.413+0000 7f92cb7fe700 1 -- 172.21.15.194:0/1682769935 learned_addr learned my addr 172.21.15.194:0/1682769935 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:56:21.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:21.413+0000 7f92cb7fe700 1 -- 172.21.15.194:0/1682769935 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c0005040 con 0x7f92cc0f5410 2024-01-18T06:56:21.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:21.413+0000 7f92cb7fe700 1 --2- 172.21.15.194:0/1682769935 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f92cc0f5410 0x7f92cc105630 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f92c0013510 tx=0x7f92c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:56:21.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:21.413+0000 7f92b3fff700 1 -- 172.21.15.194:0/1682769935 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f92c0009320 con 0x7f92cc0f5410 2024-01-18T06:56:21.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:21.413+0000 7f92b3fff700 1 -- 172.21.15.194:0/1682769935 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f92c0013f00 con 0x7f92cc0f5410 2024-01-18T06:56:21.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:21.413+0000 7f92d1d04700 1 -- 172.21.15.194:0/1682769935 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92cc105d20 con 0x7f92cc0f5410 2024-01-18T06:56:21.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:21.413+0000 7f92b3fff700 1 -- 172.21.15.194:0/1682769935 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f92c0014d90 con 0x7f92cc0f5410 2024-01-18T06:56:21.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:21.413+0000 7f92d1d04700 1 -- 172.21.15.194:0/1682769935 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92cc1061b0 con 0x7f92cc0f5410 2024-01-18T06:56:21.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:21.414+0000 7f92b3fff700 1 -- 172.21.15.194:0/1682769935 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f92c0014420 con 0x7f92cc0f5410 2024-01-18T06:56:21.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:21.414+0000 7f92d1d04700 1 -- 172.21.15.194:0/1682769935 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92cc045ad0 con 0x7f92cc0f5410 2024-01-18T06:56:21.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:21.414+0000 7f92b3fff700 1 --2- 172.21.15.194:0/1682769935 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f92b402ff60 0x7f92b4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:56:21.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:21.414+0000 7f92b3fff700 1 -- 172.21.15.194:0/1682769935 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f92c0042e70 con 0x7f92cc0f5410 2024-01-18T06:56:21.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:21.414+0000 7f92caffd700 1 --2- 172.21.15.194:0/1682769935 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f92b402ff60 0x7f92b4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:56:21.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:21.415+0000 7f92caffd700 1 --2- 172.21.15.194:0/1682769935 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f92b402ff60 0x7f92b4032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f92bc0096a0 tx=0x7f92bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:56:21.419 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:21.420+0000 7f92b3fff700 1 -- 172.21.15.194:0/1682769935 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f92c00095f0 con 0x7f92cc0f5410 2024-01-18T06:56:21.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:21 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:56:20.049580+0000 mgr.smithi047.fuhzzr (mgr.14162) 243 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:56:21.626 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:21.625+0000 7f92d1d04700 1 -- 172.21.15.194:0/1682769935 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f92cc0fae60 con 0x7f92cc0f5410 2024-01-18T06:56:21.627 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:21.626+0000 7f92b3fff700 1 -- 172.21.15.194:0/1682769935 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f92c001fb70 con 0x7f92cc0f5410 2024-01-18T06:56:21.627 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:56:21.627 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:56:21.628 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:21.629+0000 7f92d1d04700 1 -- 172.21.15.194:0/1682769935 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f92b402ff60 msgr2=0x7f92b4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:21.628 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:21.629+0000 7f92d1d04700 1 --2- 172.21.15.194:0/1682769935 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f92b402ff60 0x7f92b4032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f92bc0096a0 tx=0x7f92bc006b40).stop 2024-01-18T06:56:21.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:21.629+0000 7f92d1d04700 1 -- 172.21.15.194:0/1682769935 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f92cc0f5410 msgr2=0x7f92cc105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:21.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:21.629+0000 7f92d1d04700 1 --2- 172.21.15.194:0/1682769935 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f92cc0f5410 0x7f92cc105630 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f92c0013510 tx=0x7f92c000b040).stop 2024-01-18T06:56:21.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:21.629+0000 7f92d1d04700 1 -- 172.21.15.194:0/1682769935 shutdown_connections 2024-01-18T06:56:21.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:21.629+0000 7f92d1d04700 1 --2- 172.21.15.194:0/1682769935 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f92b402ff60 0x7f92b4032420 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:21.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:21.629+0000 7f92d1d04700 1 --2- 172.21.15.194:0/1682769935 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f92cc0f5410 0x7f92cc105630 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:21.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:21.629+0000 7f92d1d04700 1 -- 172.21.15.194:0/1682769935 >> 172.21.15.194:0/1682769935 conn(0x7f92cc0f0b70 msgr2=0x7f92cc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:56:21.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:21.629+0000 7f92d1d04700 1 -- 172.21.15.194:0/1682769935 shutdown_connections 2024-01-18T06:56:21.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:21.629+0000 7f92d1d04700 1 -- 172.21.15.194:0/1682769935 wait complete. 2024-01-18T06:56:21.629 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:56:22.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:22 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:56:21.628063+0000 mon.smithi047 (mon.0) 297 : audit [DBG] from='client.? 172.21.15.194:0/1682769935' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:56:23.037 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:56:23.037 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:56:23.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:23 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:56:22.049876+0000 mgr.smithi047.fuhzzr (mgr.14162) 244 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:56:24.278 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:24.277+0000 7f1eebd2c700 1 -- 172.21.15.194:0/1266701150 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ee40f6510 msgr2=0x7f1ee40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:24.279 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:24.277+0000 7f1eebd2c700 1 --2- 172.21.15.194:0/1266701150 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ee40f6510 0x7f1ee40f6930 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f1ed4009fb0 tx=0x7f1ed4009b30).stop 2024-01-18T06:56:24.279 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:24.278+0000 7f1eebd2c700 1 -- 172.21.15.194:0/1266701150 shutdown_connections 2024-01-18T06:56:24.279 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:24.278+0000 7f1eebd2c700 1 --2- 172.21.15.194:0/1266701150 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ee40f6510 0x7f1ee40f6930 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:24.279 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:24.278+0000 7f1eebd2c700 1 -- 172.21.15.194:0/1266701150 >> 172.21.15.194:0/1266701150 conn(0x7f1ee40f1c70 msgr2=0x7f1ee40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:56:24.279 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:24.278+0000 7f1eebd2c700 1 -- 172.21.15.194:0/1266701150 shutdown_connections 2024-01-18T06:56:24.279 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:24.278+0000 7f1eebd2c700 1 -- 172.21.15.194:0/1266701150 wait complete. 2024-01-18T06:56:24.279 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:24.278+0000 7f1eebd2c700 1 Processor -- start 2024-01-18T06:56:24.279 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:24.278+0000 7f1eebd2c700 1 -- start start 2024-01-18T06:56:24.279 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:24.278+0000 7f1eebd2c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ee40f6510 0x7f1ee4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:56:24.279 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:24.278+0000 7f1eebd2c700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee4106c20 con 0x7f1ee40f6510 2024-01-18T06:56:24.279 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:24.279+0000 7f1ee9ac8700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ee40f6510 0x7f1ee4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:56:24.279 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:24.279+0000 7f1ee9ac8700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ee40f6510 0x7f1ee4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:58026/0 (socket says 172.21.15.194:58026) 2024-01-18T06:56:24.280 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:24.279+0000 7f1ee9ac8700 1 -- 172.21.15.194:0/2655340523 learned_addr learned my addr 172.21.15.194:0/2655340523 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:56:24.280 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:24.279+0000 7f1ee9ac8700 1 -- 172.21.15.194:0/2655340523 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ed4005040 con 0x7f1ee40f6510 2024-01-18T06:56:24.280 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:24.279+0000 7f1ee9ac8700 1 --2- 172.21.15.194:0/2655340523 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ee40f6510 0x7f1ee4106730 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f1ed4013510 tx=0x7f1ed400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:56:24.280 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:24.279+0000 7f1eda7fc700 1 -- 172.21.15.194:0/2655340523 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ed4009320 con 0x7f1ee40f6510 2024-01-18T06:56:24.280 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:24.279+0000 7f1eebd2c700 1 -- 172.21.15.194:0/2655340523 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ee4106e20 con 0x7f1ee40f6510 2024-01-18T06:56:24.280 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:24.279+0000 7f1eda7fc700 1 -- 172.21.15.194:0/2655340523 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1ed4013f00 con 0x7f1ee40f6510 2024-01-18T06:56:24.280 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:24.279+0000 7f1eda7fc700 1 -- 172.21.15.194:0/2655340523 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ed4014d90 con 0x7f1ee40f6510 2024-01-18T06:56:24.280 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:24.279+0000 7f1eebd2c700 1 -- 172.21.15.194:0/2655340523 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ee41072b0 con 0x7f1ee40f6510 2024-01-18T06:56:24.281 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:24.280+0000 7f1eda7fc700 1 -- 172.21.15.194:0/2655340523 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1ed40263a0 con 0x7f1ee40f6510 2024-01-18T06:56:24.281 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:24.280+0000 7f1eda7fc700 1 --2- 172.21.15.194:0/2655340523 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f1ed002ffb0 0x7f1ed0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:56:24.281 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:24.280+0000 7f1eebd2c700 1 -- 172.21.15.194:0/2655340523 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ee4045ad0 con 0x7f1ee40f6510 2024-01-18T06:56:24.281 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:24.280+0000 7f1eda7fc700 1 -- 172.21.15.194:0/2655340523 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1ed40413c0 con 0x7f1ee40f6510 2024-01-18T06:56:24.281 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:24.280+0000 7f1ee92c7700 1 --2- 172.21.15.194:0/2655340523 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f1ed002ffb0 0x7f1ed0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:56:24.281 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:24.281+0000 7f1ee92c7700 1 --2- 172.21.15.194:0/2655340523 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f1ed002ffb0 0x7f1ed0032470 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f1ee00096a0 tx=0x7f1ee0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:56:24.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:24.285+0000 7f1eda7fc700 1 -- 172.21.15.194:0/2655340523 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1ed4012330 con 0x7f1ee40f6510 2024-01-18T06:56:24.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:24.489+0000 7f1eebd2c700 1 -- 172.21.15.194:0/2655340523 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1ee40fc050 con 0x7f1ee40f6510 2024-01-18T06:56:24.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:24.490+0000 7f1eda7fc700 1 -- 172.21.15.194:0/2655340523 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1ed402b030 con 0x7f1ee40f6510 2024-01-18T06:56:24.490 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:56:24.490 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:56:24.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:24.492+0000 7f1eebd2c700 1 -- 172.21.15.194:0/2655340523 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f1ed002ffb0 msgr2=0x7f1ed0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:24.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:24.492+0000 7f1eebd2c700 1 --2- 172.21.15.194:0/2655340523 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f1ed002ffb0 0x7f1ed0032470 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f1ee00096a0 tx=0x7f1ee0006b40).stop 2024-01-18T06:56:24.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:24.492+0000 7f1eebd2c700 1 -- 172.21.15.194:0/2655340523 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ee40f6510 msgr2=0x7f1ee4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:24.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:24.492+0000 7f1eebd2c700 1 --2- 172.21.15.194:0/2655340523 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ee40f6510 0x7f1ee4106730 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f1ed4013510 tx=0x7f1ed400b040).stop 2024-01-18T06:56:24.492 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:24.492+0000 7f1eebd2c700 1 -- 172.21.15.194:0/2655340523 shutdown_connections 2024-01-18T06:56:24.492 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:24.492+0000 7f1eebd2c700 1 --2- 172.21.15.194:0/2655340523 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f1ed002ffb0 0x7f1ed0032470 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:24.492 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:24.492+0000 7f1eebd2c700 1 --2- 172.21.15.194:0/2655340523 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1ee40f6510 0x7f1ee4106730 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:24.492 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:24.492+0000 7f1eebd2c700 1 -- 172.21.15.194:0/2655340523 >> 172.21.15.194:0/2655340523 conn(0x7f1ee40f1c70 msgr2=0x7f1ee40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:56:24.492 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:24.492+0000 7f1eebd2c700 1 -- 172.21.15.194:0/2655340523 shutdown_connections 2024-01-18T06:56:24.492 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:24.492+0000 7f1eebd2c700 1 -- 172.21.15.194:0/2655340523 wait complete. 2024-01-18T06:56:24.492 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:56:25.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:25 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:56:24.050128+0000 mgr.smithi047.fuhzzr (mgr.14162) 245 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:56:25.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:25 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:56:24.491090+0000 mon.smithi047 (mon.0) 298 : audit [DBG] from='client.? 172.21.15.194:0/2655340523' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:56:25.869 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:56:25.869 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:56:27.020 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:27.020+0000 7f5c5f00e700 1 -- 172.21.15.194:0/4149775512 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c580f5410 msgr2=0x7f5c580f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:27.020 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:27.020+0000 7f5c5f00e700 1 --2- 172.21.15.194:0/4149775512 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c580f5410 0x7f5c580f5830 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f5c4c009fb0 tx=0x7f5c4c009b30).stop 2024-01-18T06:56:27.020 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:27.020+0000 7f5c5f00e700 1 -- 172.21.15.194:0/4149775512 shutdown_connections 2024-01-18T06:56:27.020 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:27.020+0000 7f5c5f00e700 1 --2- 172.21.15.194:0/4149775512 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c580f5410 0x7f5c580f5830 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:27.020 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:27.020+0000 7f5c5f00e700 1 -- 172.21.15.194:0/4149775512 >> 172.21.15.194:0/4149775512 conn(0x7f5c580f0b70 msgr2=0x7f5c580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:56:27.020 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:27.020+0000 7f5c5f00e700 1 -- 172.21.15.194:0/4149775512 shutdown_connections 2024-01-18T06:56:27.020 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:27.020+0000 7f5c5f00e700 1 -- 172.21.15.194:0/4149775512 wait complete. 2024-01-18T06:56:27.022 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:27.021+0000 7f5c5f00e700 1 Processor -- start 2024-01-18T06:56:27.022 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:27.021+0000 7f5c5f00e700 1 -- start start 2024-01-18T06:56:27.022 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:27.021+0000 7f5c5f00e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c580f5410 0x7f5c58105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:56:27.022 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:27.021+0000 7f5c5f00e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c58105b20 con 0x7f5c580f5410 2024-01-18T06:56:27.022 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:27.021+0000 7f5c5cdaa700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c580f5410 0x7f5c58105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:56:27.022 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:27.021+0000 7f5c5cdaa700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c580f5410 0x7f5c58105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:58040/0 (socket says 172.21.15.194:58040) 2024-01-18T06:56:27.022 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:27.022+0000 7f5c5cdaa700 1 -- 172.21.15.194:0/3681979060 learned_addr learned my addr 172.21.15.194:0/3681979060 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:56:27.022 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:27.022+0000 7f5c5cdaa700 1 -- 172.21.15.194:0/3681979060 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c4c005040 con 0x7f5c580f5410 2024-01-18T06:56:27.022 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:27.022+0000 7f5c5cdaa700 1 --2- 172.21.15.194:0/3681979060 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c580f5410 0x7f5c58105630 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f5c4c013510 tx=0x7f5c4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:56:27.022 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:27.022+0000 7f5c557fa700 1 -- 172.21.15.194:0/3681979060 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c4c009320 con 0x7f5c580f5410 2024-01-18T06:56:27.023 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:27.022+0000 7f5c5f00e700 1 -- 172.21.15.194:0/3681979060 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c58105d20 con 0x7f5c580f5410 2024-01-18T06:56:27.023 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:27.022+0000 7f5c557fa700 1 -- 172.21.15.194:0/3681979060 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5c4c013f00 con 0x7f5c580f5410 2024-01-18T06:56:27.023 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:27.022+0000 7f5c557fa700 1 -- 172.21.15.194:0/3681979060 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c4c014d90 con 0x7f5c580f5410 2024-01-18T06:56:27.023 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:27.022+0000 7f5c5f00e700 1 -- 172.21.15.194:0/3681979060 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c581061b0 con 0x7f5c580f5410 2024-01-18T06:56:27.024 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:27.023+0000 7f5c557fa700 1 -- 172.21.15.194:0/3681979060 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5c4c014420 con 0x7f5c580f5410 2024-01-18T06:56:27.024 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:27.023+0000 7f5c5f00e700 1 -- 172.21.15.194:0/3681979060 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c58045ad0 con 0x7f5c580f5410 2024-01-18T06:56:27.024 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:27.023+0000 7f5c557fa700 1 --2- 172.21.15.194:0/3681979060 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f5c4802ff60 0x7f5c48032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:56:27.024 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:27.023+0000 7f5c557fa700 1 -- 172.21.15.194:0/3681979060 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5c4c041070 con 0x7f5c580f5410 2024-01-18T06:56:27.024 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:27.023+0000 7f5c57fff700 1 --2- 172.21.15.194:0/3681979060 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f5c4802ff60 0x7f5c48032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:56:27.024 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:27.024+0000 7f5c57fff700 1 --2- 172.21.15.194:0/3681979060 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f5c4802ff60 0x7f5c48032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f5c440096a0 tx=0x7f5c44006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:56:27.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:27.028+0000 7f5c557fa700 1 -- 172.21.15.194:0/3681979060 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5c4c0095f0 con 0x7f5c580f5410 2024-01-18T06:56:27.234 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:27.234+0000 7f5c5f00e700 1 -- 172.21.15.194:0/3681979060 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5c580fae60 con 0x7f5c580f5410 2024-01-18T06:56:27.235 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:27.235+0000 7f5c557fa700 1 -- 172.21.15.194:0/3681979060 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5c4c01db70 con 0x7f5c580f5410 2024-01-18T06:56:27.235 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:56:27.235 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:56:27.236 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:27.237+0000 7f5c5f00e700 1 -- 172.21.15.194:0/3681979060 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f5c4802ff60 msgr2=0x7f5c48032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:27.236 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:27.237+0000 7f5c5f00e700 1 --2- 172.21.15.194:0/3681979060 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f5c4802ff60 0x7f5c48032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f5c440096a0 tx=0x7f5c44006b40).stop 2024-01-18T06:56:27.236 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:27.237+0000 7f5c5f00e700 1 -- 172.21.15.194:0/3681979060 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c580f5410 msgr2=0x7f5c58105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:27.236 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:27.237+0000 7f5c5f00e700 1 --2- 172.21.15.194:0/3681979060 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c580f5410 0x7f5c58105630 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f5c4c013510 tx=0x7f5c4c00b040).stop 2024-01-18T06:56:27.237 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:27.237+0000 7f5c5f00e700 1 -- 172.21.15.194:0/3681979060 shutdown_connections 2024-01-18T06:56:27.237 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:27.237+0000 7f5c5f00e700 1 --2- 172.21.15.194:0/3681979060 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f5c4802ff60 0x7f5c48032420 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:27.237 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:27.237+0000 7f5c5f00e700 1 --2- 172.21.15.194:0/3681979060 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5c580f5410 0x7f5c58105630 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:27.237 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:27.237+0000 7f5c5f00e700 1 -- 172.21.15.194:0/3681979060 >> 172.21.15.194:0/3681979060 conn(0x7f5c580f0b70 msgr2=0x7f5c580f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:56:27.237 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:27.237+0000 7f5c5f00e700 1 -- 172.21.15.194:0/3681979060 shutdown_connections 2024-01-18T06:56:27.237 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:27.237+0000 7f5c5f00e700 1 -- 172.21.15.194:0/3681979060 wait complete. 2024-01-18T06:56:27.237 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:56:27.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:27 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:56:26.050481+0000 mgr.smithi047.fuhzzr (mgr.14162) 246 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:56:28.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:28 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:56:27.236197+0000 mon.smithi047 (mon.0) 299 : audit [DBG] from='client.? 172.21.15.194:0/3681979060' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:56:28.628 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:56:28.628 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:56:29.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:29 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:56:28.050879+0000 mgr.smithi047.fuhzzr (mgr.14162) 247 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:56:29.838 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:29.837+0000 7fc71e503700 1 -- 172.21.15.194:0/1944455670 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7180f6510 msgr2=0x7fc7180f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:29.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:29.837+0000 7fc71e503700 1 --2- 172.21.15.194:0/1944455670 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7180f6510 0x7fc7180f6930 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7fc708009fb0 tx=0x7fc708009b30).stop 2024-01-18T06:56:29.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:29.837+0000 7fc71e503700 1 -- 172.21.15.194:0/1944455670 shutdown_connections 2024-01-18T06:56:29.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:29.837+0000 7fc71e503700 1 --2- 172.21.15.194:0/1944455670 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7180f6510 0x7fc7180f6930 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:29.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:29.837+0000 7fc71e503700 1 -- 172.21.15.194:0/1944455670 >> 172.21.15.194:0/1944455670 conn(0x7fc7180f1c70 msgr2=0x7fc7180f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:56:29.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:29.837+0000 7fc71e503700 1 -- 172.21.15.194:0/1944455670 shutdown_connections 2024-01-18T06:56:29.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:29.837+0000 7fc71e503700 1 -- 172.21.15.194:0/1944455670 wait complete. 2024-01-18T06:56:29.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:29.837+0000 7fc71e503700 1 Processor -- start 2024-01-18T06:56:29.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:29.838+0000 7fc71e503700 1 -- start start 2024-01-18T06:56:29.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:29.838+0000 7fc71e503700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7180f6510 0x7fc71806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:56:29.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:29.838+0000 7fc71e503700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc718069e20 con 0x7fc7180f6510 2024-01-18T06:56:29.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:29.838+0000 7fc717fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7180f6510 0x7fc71806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:56:29.840 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:29.838+0000 7fc717fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7180f6510 0x7fc71806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:52912/0 (socket says 172.21.15.194:52912) 2024-01-18T06:56:29.840 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:29.838+0000 7fc717fff700 1 -- 172.21.15.194:0/1530405652 learned_addr learned my addr 172.21.15.194:0/1530405652 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:56:29.840 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:29.838+0000 7fc717fff700 1 -- 172.21.15.194:0/1530405652 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc708005040 con 0x7fc7180f6510 2024-01-18T06:56:29.840 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:29.838+0000 7fc717fff700 1 --2- 172.21.15.194:0/1530405652 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7180f6510 0x7fc71806b9a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fc708013510 tx=0x7fc70800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:56:29.840 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:29.839+0000 7fc714ff9700 1 -- 172.21.15.194:0/1530405652 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc708009320 con 0x7fc7180f6510 2024-01-18T06:56:29.840 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:29.839+0000 7fc714ff9700 1 -- 172.21.15.194:0/1530405652 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc708013f00 con 0x7fc7180f6510 2024-01-18T06:56:29.840 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:29.839+0000 7fc714ff9700 1 -- 172.21.15.194:0/1530405652 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc708014d90 con 0x7fc7180f6510 2024-01-18T06:56:29.840 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:29.839+0000 7fc71e503700 1 -- 172.21.15.194:0/1530405652 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc718069fc0 con 0x7fc7180f6510 2024-01-18T06:56:29.840 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:29.839+0000 7fc71e503700 1 -- 172.21.15.194:0/1530405652 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc71806a450 con 0x7fc7180f6510 2024-01-18T06:56:29.841 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:29.839+0000 7fc714ff9700 1 -- 172.21.15.194:0/1530405652 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc7080263a0 con 0x7fc7180f6510 2024-01-18T06:56:29.841 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:29.840+0000 7fc714ff9700 1 --2- 172.21.15.194:0/1530405652 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc70002ff60 0x7fc700032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:56:29.841 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:29.840+0000 7fc714ff9700 1 -- 172.21.15.194:0/1530405652 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc7080412f0 con 0x7fc7180f6510 2024-01-18T06:56:29.841 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:29.840+0000 7fc71e503700 1 -- 172.21.15.194:0/1530405652 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc718059100 con 0x7fc7180f6510 2024-01-18T06:56:29.841 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:29.840+0000 7fc7177fe700 1 --2- 172.21.15.194:0/1530405652 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc70002ff60 0x7fc700032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:56:29.841 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:29.840+0000 7fc7177fe700 1 --2- 172.21.15.194:0/1530405652 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc70002ff60 0x7fc700032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fc70c0096a0 tx=0x7fc70c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:56:29.844 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:29.845+0000 7fc714ff9700 1 -- 172.21.15.194:0/1530405652 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc70801cad0 con 0x7fc7180f6510 2024-01-18T06:56:30.050 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:30.051+0000 7fc71e503700 1 -- 172.21.15.194:0/1530405652 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc71806ab50 con 0x7fc7180f6510 2024-01-18T06:56:30.051 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:30.052+0000 7fc714ff9700 1 -- 172.21.15.194:0/1530405652 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc70802b030 con 0x7fc7180f6510 2024-01-18T06:56:30.051 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:56:30.052 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:56:30.053 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:30.053+0000 7fc71e503700 1 -- 172.21.15.194:0/1530405652 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc70002ff60 msgr2=0x7fc700032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:30.053 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:30.053+0000 7fc71e503700 1 --2- 172.21.15.194:0/1530405652 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc70002ff60 0x7fc700032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fc70c0096a0 tx=0x7fc70c006b40).stop 2024-01-18T06:56:30.053 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:30.053+0000 7fc71e503700 1 -- 172.21.15.194:0/1530405652 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7180f6510 msgr2=0x7fc71806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:30.053 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:30.054+0000 7fc71e503700 1 --2- 172.21.15.194:0/1530405652 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7180f6510 0x7fc71806b9a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fc708013510 tx=0x7fc70800b040).stop 2024-01-18T06:56:30.053 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:30.054+0000 7fc71e503700 1 -- 172.21.15.194:0/1530405652 shutdown_connections 2024-01-18T06:56:30.054 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:30.054+0000 7fc71e503700 1 --2- 172.21.15.194:0/1530405652 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc70002ff60 0x7fc700032420 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:30.054 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:30.054+0000 7fc71e503700 1 --2- 172.21.15.194:0/1530405652 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7180f6510 0x7fc71806b9a0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:30.054 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:30.054+0000 7fc71e503700 1 -- 172.21.15.194:0/1530405652 >> 172.21.15.194:0/1530405652 conn(0x7fc7180f1c70 msgr2=0x7fc7180f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:56:30.054 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:30.054+0000 7fc71e503700 1 -- 172.21.15.194:0/1530405652 shutdown_connections 2024-01-18T06:56:30.054 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:30.054+0000 7fc71e503700 1 -- 172.21.15.194:0/1530405652 wait complete. 2024-01-18T06:56:30.054 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:56:30.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:30 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:56:30.052827+0000 mon.smithi047 (mon.0) 300 : audit [DBG] from='client.? 172.21.15.194:0/1530405652' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:56:31.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:31 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:56:30.051237+0000 mgr.smithi047.fuhzzr (mgr.14162) 248 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:56:31.213 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:31 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:56:30.119883+0000 mon.smithi047 (mon.0) 301 : audit [INF] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.fuhzzr/mirror_snapshot_schedule"}]: dispatch 2024-01-18T06:56:31.214 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:31 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:56:30.120100+0000 mon.smithi047 (mon.0) 302 : audit [INF] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.fuhzzr/trash_purge_schedule"}]: dispatch 2024-01-18T06:56:31.429 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:56:31.430 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:56:32.566 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:32.565+0000 7f1c257eb700 1 -- 172.21.15.194:0/160682637 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c200f5410 msgr2=0x7f1c200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:32.566 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:32.565+0000 7f1c257eb700 1 --2- 172.21.15.194:0/160682637 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c200f5410 0x7f1c200f5830 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f1c14009fb0 tx=0x7f1c14009b30).stop 2024-01-18T06:56:32.566 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:32.565+0000 7f1c257eb700 1 -- 172.21.15.194:0/160682637 shutdown_connections 2024-01-18T06:56:32.566 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:32.565+0000 7f1c257eb700 1 --2- 172.21.15.194:0/160682637 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c200f5410 0x7f1c200f5830 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:32.566 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:32.565+0000 7f1c257eb700 1 -- 172.21.15.194:0/160682637 >> 172.21.15.194:0/160682637 conn(0x7f1c200f0b70 msgr2=0x7f1c200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:56:32.566 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:32.565+0000 7f1c257eb700 1 -- 172.21.15.194:0/160682637 shutdown_connections 2024-01-18T06:56:32.566 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:32.565+0000 7f1c257eb700 1 -- 172.21.15.194:0/160682637 wait complete. 2024-01-18T06:56:32.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:32.566+0000 7f1c257eb700 1 Processor -- start 2024-01-18T06:56:32.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:32.566+0000 7f1c257eb700 1 -- start start 2024-01-18T06:56:32.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:32.566+0000 7f1c257eb700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c200601e0 0x7f1c2010b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:56:32.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:32.566+0000 7f1c257eb700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c20107390 con 0x7f1c200601e0 2024-01-18T06:56:32.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:32.566+0000 7f1c1effd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c200601e0 0x7f1c2010b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:56:32.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:32.566+0000 7f1c1effd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c200601e0 0x7f1c2010b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:52934/0 (socket says 172.21.15.194:52934) 2024-01-18T06:56:32.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:32.566+0000 7f1c1effd700 1 -- 172.21.15.194:0/3626280239 learned_addr learned my addr 172.21.15.194:0/3626280239 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:56:32.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:32.566+0000 7f1c1effd700 1 -- 172.21.15.194:0/3626280239 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c14005040 con 0x7f1c200601e0 2024-01-18T06:56:32.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:32.566+0000 7f1c1effd700 1 --2- 172.21.15.194:0/3626280239 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c200601e0 0x7f1c2010b9a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f1c14009f80 tx=0x7f1c14009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:56:32.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:32.567+0000 7f1c077fe700 1 -- 172.21.15.194:0/3626280239 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c140136c0 con 0x7f1c200601e0 2024-01-18T06:56:32.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:32.567+0000 7f1c077fe700 1 -- 172.21.15.194:0/3626280239 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1c14013e70 con 0x7f1c200601e0 2024-01-18T06:56:32.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:32.567+0000 7f1c257eb700 1 -- 172.21.15.194:0/3626280239 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c2010be90 con 0x7f1c200601e0 2024-01-18T06:56:32.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:32.567+0000 7f1c077fe700 1 -- 172.21.15.194:0/3626280239 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c14014420 con 0x7f1c200601e0 2024-01-18T06:56:32.568 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:32.567+0000 7f1c257eb700 1 -- 172.21.15.194:0/3626280239 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c2010c1e0 con 0x7f1c200601e0 2024-01-18T06:56:32.568 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:32.567+0000 7f1c077fe700 1 -- 172.21.15.194:0/3626280239 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1c14025060 con 0x7f1c200601e0 2024-01-18T06:56:32.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:32.568+0000 7f1c077fe700 1 --2- 172.21.15.194:0/3626280239 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f1c0802ff60 0x7f1c08032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:56:32.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:32.568+0000 7f1c077fe700 1 -- 172.21.15.194:0/3626280239 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1c14040e30 con 0x7f1c200601e0 2024-01-18T06:56:32.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:32.568+0000 7f1c257eb700 1 -- 172.21.15.194:0/3626280239 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c20045ad0 con 0x7f1c200601e0 2024-01-18T06:56:32.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:32.568+0000 7f1c1e7fc700 1 --2- 172.21.15.194:0/3626280239 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f1c0802ff60 0x7f1c08032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:56:32.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:32.568+0000 7f1c1e7fc700 1 --2- 172.21.15.194:0/3626280239 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f1c0802ff60 0x7f1c08032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f1c100096a0 tx=0x7f1c10006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:56:32.572 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:32.573+0000 7f1c077fe700 1 -- 172.21.15.194:0/3626280239 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1c14028440 con 0x7f1c200601e0 2024-01-18T06:56:32.778 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:32.779+0000 7f1c257eb700 1 -- 172.21.15.194:0/3626280239 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1c2010ca10 con 0x7f1c200601e0 2024-01-18T06:56:32.779 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:32.780+0000 7f1c077fe700 1 -- 172.21.15.194:0/3626280239 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1c1401c340 con 0x7f1c200601e0 2024-01-18T06:56:32.779 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:56:32.779 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:56:32.781 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:32.781+0000 7f1c257eb700 1 -- 172.21.15.194:0/3626280239 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f1c0802ff60 msgr2=0x7f1c08032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:32.781 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:32.781+0000 7f1c257eb700 1 --2- 172.21.15.194:0/3626280239 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f1c0802ff60 0x7f1c08032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f1c100096a0 tx=0x7f1c10006b40).stop 2024-01-18T06:56:32.781 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:32.781+0000 7f1c257eb700 1 -- 172.21.15.194:0/3626280239 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c200601e0 msgr2=0x7f1c2010b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:32.781 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:32.781+0000 7f1c257eb700 1 --2- 172.21.15.194:0/3626280239 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c200601e0 0x7f1c2010b9a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f1c14009f80 tx=0x7f1c14009b30).stop 2024-01-18T06:56:32.781 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:32.781+0000 7f1c257eb700 1 -- 172.21.15.194:0/3626280239 shutdown_connections 2024-01-18T06:56:32.781 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:32.781+0000 7f1c257eb700 1 --2- 172.21.15.194:0/3626280239 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f1c0802ff60 0x7f1c08032420 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:32.781 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:32.781+0000 7f1c257eb700 1 --2- 172.21.15.194:0/3626280239 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c200601e0 0x7f1c2010b9a0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:32.781 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:32.781+0000 7f1c257eb700 1 -- 172.21.15.194:0/3626280239 >> 172.21.15.194:0/3626280239 conn(0x7f1c200f0b70 msgr2=0x7f1c200f1700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:56:32.781 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:32.781+0000 7f1c257eb700 1 -- 172.21.15.194:0/3626280239 shutdown_connections 2024-01-18T06:56:32.781 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:32.781+0000 7f1c257eb700 1 -- 172.21.15.194:0/3626280239 wait complete. 2024-01-18T06:56:32.782 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:56:33.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:33 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:56:32.051552+0000 mgr.smithi047.fuhzzr (mgr.14162) 249 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:56:33.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:33 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:56:32.780597+0000 mon.smithi047 (mon.0) 303 : audit [DBG] from='client.? 172.21.15.194:0/3626280239' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:56:34.159 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:56:34.159 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:56:35.354 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:35.353+0000 7fdcae9f6700 1 -- 172.21.15.194:0/1033592151 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdca80f47c0 msgr2=0x7fdca80f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:35.354 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:35.353+0000 7fdcae9f6700 1 --2- 172.21.15.194:0/1033592151 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdca80f47c0 0x7fdca80f4be0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7fdc9c009fb0 tx=0x7fdc9c009b30).stop 2024-01-18T06:56:35.354 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:35.353+0000 7fdcae9f6700 1 -- 172.21.15.194:0/1033592151 shutdown_connections 2024-01-18T06:56:35.354 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:35.353+0000 7fdcae9f6700 1 --2- 172.21.15.194:0/1033592151 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdca80f47c0 0x7fdca80f4be0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:35.354 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:35.353+0000 7fdcae9f6700 1 -- 172.21.15.194:0/1033592151 >> 172.21.15.194:0/1033592151 conn(0x7fdca80f02e0 msgr2=0x7fdca80f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:56:35.355 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:35.353+0000 7fdcae9f6700 1 -- 172.21.15.194:0/1033592151 shutdown_connections 2024-01-18T06:56:35.355 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:35.353+0000 7fdcae9f6700 1 -- 172.21.15.194:0/1033592151 wait complete. 2024-01-18T06:56:35.355 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:35.354+0000 7fdcae9f6700 1 Processor -- start 2024-01-18T06:56:35.355 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:35.354+0000 7fdcae9f6700 1 -- start start 2024-01-18T06:56:35.355 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:35.354+0000 7fdcae9f6700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdca80f47c0 0x7fdca80f7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:56:35.355 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:35.354+0000 7fdcae9f6700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca80f57f0 con 0x7fdca80f47c0 2024-01-18T06:56:35.355 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:35.354+0000 7fdca7fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdca80f47c0 0x7fdca80f7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:56:35.355 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:35.354+0000 7fdca7fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdca80f47c0 0x7fdca80f7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:52956/0 (socket says 172.21.15.194:52956) 2024-01-18T06:56:35.355 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:35.354+0000 7fdca7fff700 1 -- 172.21.15.194:0/3642428466 learned_addr learned my addr 172.21.15.194:0/3642428466 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:56:35.355 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:35.354+0000 7fdca7fff700 1 -- 172.21.15.194:0/3642428466 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc9c005040 con 0x7fdca80f47c0 2024-01-18T06:56:35.355 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:35.355+0000 7fdca7fff700 1 --2- 172.21.15.194:0/3642428466 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdca80f47c0 0x7fdca80f7370 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fdc9c013510 tx=0x7fdc9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:56:35.355 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:35.355+0000 7fdca4ff9700 1 -- 172.21.15.194:0/3642428466 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc9c009320 con 0x7fdca80f47c0 2024-01-18T06:56:35.355 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:35.355+0000 7fdcae9f6700 1 -- 172.21.15.194:0/3642428466 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdca80f5990 con 0x7fdca80f47c0 2024-01-18T06:56:35.355 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:35.355+0000 7fdca4ff9700 1 -- 172.21.15.194:0/3642428466 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdc9c013f00 con 0x7fdca80f47c0 2024-01-18T06:56:35.356 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:35.355+0000 7fdca4ff9700 1 -- 172.21.15.194:0/3642428466 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc9c014d90 con 0x7fdca80f47c0 2024-01-18T06:56:35.356 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:35.355+0000 7fdcae9f6700 1 -- 172.21.15.194:0/3642428466 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdca80f5e20 con 0x7fdca80f47c0 2024-01-18T06:56:35.357 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:35.356+0000 7fdca4ff9700 1 -- 172.21.15.194:0/3642428466 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdc9c014420 con 0x7fdca80f47c0 2024-01-18T06:56:35.357 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:35.356+0000 7fdcae9f6700 1 -- 172.21.15.194:0/3642428466 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdca8059100 con 0x7fdca80f47c0 2024-01-18T06:56:35.357 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:35.356+0000 7fdca4ff9700 1 --2- 172.21.15.194:0/3642428466 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fdc9002ff60 0x7fdc90032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:56:35.357 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:35.356+0000 7fdca4ff9700 1 -- 172.21.15.194:0/3642428466 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdc9c0410e0 con 0x7fdca80f47c0 2024-01-18T06:56:35.357 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:35.356+0000 7fdca77fe700 1 --2- 172.21.15.194:0/3642428466 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fdc9002ff60 0x7fdc90032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:56:35.357 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:35.356+0000 7fdca77fe700 1 --2- 172.21.15.194:0/3642428466 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fdc9002ff60 0x7fdc90032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fdc980096a0 tx=0x7fdc98006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:56:35.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:35.361+0000 7fdca4ff9700 1 -- 172.21.15.194:0/3642428466 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdc9c0095f0 con 0x7fdca80f47c0 2024-01-18T06:56:35.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:35 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:56:34.051853+0000 mgr.smithi047.fuhzzr (mgr.14162) 250 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:56:35.566 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:35.566+0000 7fdcae9f6700 1 -- 172.21.15.194:0/3642428466 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdca8045ad0 con 0x7fdca80f47c0 2024-01-18T06:56:35.566 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:35.567+0000 7fdca4ff9700 1 -- 172.21.15.194:0/3642428466 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdc9c01db70 con 0x7fdca80f47c0 2024-01-18T06:56:35.567 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:56:35.567 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:56:35.568 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:35.569+0000 7fdcae9f6700 1 -- 172.21.15.194:0/3642428466 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fdc9002ff60 msgr2=0x7fdc90032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:35.568 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:35.569+0000 7fdcae9f6700 1 --2- 172.21.15.194:0/3642428466 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fdc9002ff60 0x7fdc90032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fdc980096a0 tx=0x7fdc98006b40).stop 2024-01-18T06:56:35.568 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:35.569+0000 7fdcae9f6700 1 -- 172.21.15.194:0/3642428466 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdca80f47c0 msgr2=0x7fdca80f7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:35.568 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:35.569+0000 7fdcae9f6700 1 --2- 172.21.15.194:0/3642428466 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdca80f47c0 0x7fdca80f7370 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fdc9c013510 tx=0x7fdc9c00b040).stop 2024-01-18T06:56:35.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:35.569+0000 7fdcae9f6700 1 -- 172.21.15.194:0/3642428466 shutdown_connections 2024-01-18T06:56:35.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:35.569+0000 7fdcae9f6700 1 --2- 172.21.15.194:0/3642428466 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fdc9002ff60 0x7fdc90032420 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:35.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:35.569+0000 7fdcae9f6700 1 --2- 172.21.15.194:0/3642428466 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdca80f47c0 0x7fdca80f7370 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:35.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:35.569+0000 7fdcae9f6700 1 -- 172.21.15.194:0/3642428466 >> 172.21.15.194:0/3642428466 conn(0x7fdca80f02e0 msgr2=0x7fdca80f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:56:35.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:35.569+0000 7fdcae9f6700 1 -- 172.21.15.194:0/3642428466 shutdown_connections 2024-01-18T06:56:35.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:35.569+0000 7fdcae9f6700 1 -- 172.21.15.194:0/3642428466 wait complete. 2024-01-18T06:56:35.569 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:56:36.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:36 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:56:35.568211+0000 mon.smithi047 (mon.0) 304 : audit [DBG] from='client.? 172.21.15.194:0/3642428466' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:56:36.949 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:56:36.949 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:56:37.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:37 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:56:36.052166+0000 mgr.smithi047.fuhzzr (mgr.14162) 251 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:56:38.102 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:38.102+0000 7ff8162e1700 1 -- 172.21.15.194:0/3687684443 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff8100f5410 msgr2=0x7ff8100f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:38.102 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:38.102+0000 7ff8162e1700 1 --2- 172.21.15.194:0/3687684443 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff8100f5410 0x7ff8100f5830 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7ff800009fb0 tx=0x7ff800009b30).stop 2024-01-18T06:56:38.102 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:38.102+0000 7ff8162e1700 1 -- 172.21.15.194:0/3687684443 shutdown_connections 2024-01-18T06:56:38.102 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:38.102+0000 7ff8162e1700 1 --2- 172.21.15.194:0/3687684443 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff8100f5410 0x7ff8100f5830 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:38.102 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:38.102+0000 7ff8162e1700 1 -- 172.21.15.194:0/3687684443 >> 172.21.15.194:0/3687684443 conn(0x7ff8100f0b70 msgr2=0x7ff8100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:56:38.102 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:38.102+0000 7ff8162e1700 1 -- 172.21.15.194:0/3687684443 shutdown_connections 2024-01-18T06:56:38.102 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:38.102+0000 7ff8162e1700 1 -- 172.21.15.194:0/3687684443 wait complete. 2024-01-18T06:56:38.102 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:38.102+0000 7ff8162e1700 1 Processor -- start 2024-01-18T06:56:38.102 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:38.102+0000 7ff8162e1700 1 -- start start 2024-01-18T06:56:38.103 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:38.103+0000 7ff8162e1700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff8100f5410 0x7ff8100fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:56:38.103 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:38.103+0000 7ff8162e1700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8100fb820 con 0x7ff8100f5410 2024-01-18T06:56:38.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:38.103+0000 7ff80ffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff8100f5410 0x7ff8100fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:56:38.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:38.103+0000 7ff80ffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff8100f5410 0x7ff8100fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:43242/0 (socket says 172.21.15.194:43242) 2024-01-18T06:56:38.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:38.103+0000 7ff80ffff700 1 -- 172.21.15.194:0/1373021124 learned_addr learned my addr 172.21.15.194:0/1373021124 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:56:38.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:38.103+0000 7ff80ffff700 1 -- 172.21.15.194:0/1373021124 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff800005040 con 0x7ff8100f5410 2024-01-18T06:56:38.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:38.103+0000 7ff80ffff700 1 --2- 172.21.15.194:0/1373021124 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff8100f5410 0x7ff8100fd3a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7ff800013510 tx=0x7ff80000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:56:38.105 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:38.103+0000 7ff80cff9700 1 -- 172.21.15.194:0/1373021124 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff800009320 con 0x7ff8100f5410 2024-01-18T06:56:38.105 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:38.104+0000 7ff8162e1700 1 -- 172.21.15.194:0/1373021124 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8100fb9c0 con 0x7ff8100f5410 2024-01-18T06:56:38.105 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:38.104+0000 7ff80cff9700 1 -- 172.21.15.194:0/1373021124 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff800013f00 con 0x7ff8100f5410 2024-01-18T06:56:38.105 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:38.104+0000 7ff80cff9700 1 -- 172.21.15.194:0/1373021124 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff800014d90 con 0x7ff8100f5410 2024-01-18T06:56:38.105 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:38.104+0000 7ff8162e1700 1 -- 172.21.15.194:0/1373021124 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8100fbe50 con 0x7ff8100f5410 2024-01-18T06:56:38.105 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:38.104+0000 7ff80cff9700 1 -- 172.21.15.194:0/1373021124 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff800014420 con 0x7ff8100f5410 2024-01-18T06:56:38.105 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:38.104+0000 7ff8162e1700 1 -- 172.21.15.194:0/1373021124 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff810059100 con 0x7ff8100f5410 2024-01-18T06:56:38.105 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:38.105+0000 7ff80cff9700 1 --2- 172.21.15.194:0/1373021124 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7ff7fc02ff60 0x7ff7fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:56:38.105 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:38.105+0000 7ff80cff9700 1 -- 172.21.15.194:0/1373021124 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff800041070 con 0x7ff8100f5410 2024-01-18T06:56:38.105 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:38.105+0000 7ff80f7fe700 1 --2- 172.21.15.194:0/1373021124 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7ff7fc02ff60 0x7ff7fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:56:38.105 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:38.105+0000 7ff80f7fe700 1 --2- 172.21.15.194:0/1373021124 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7ff7fc02ff60 0x7ff7fc032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7ff7f80096a0 tx=0x7ff7f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:56:38.109 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:38.110+0000 7ff80cff9700 1 -- 172.21.15.194:0/1373021124 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff8000095f0 con 0x7ff8100f5410 2024-01-18T06:56:38.319 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:38.319+0000 7ff8162e1700 1 -- 172.21.15.194:0/1373021124 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff8100fae60 con 0x7ff8100f5410 2024-01-18T06:56:38.319 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:38.320+0000 7ff80cff9700 1 -- 172.21.15.194:0/1373021124 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff80001db70 con 0x7ff8100f5410 2024-01-18T06:56:38.320 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:56:38.320 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:56:38.321 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:38.322+0000 7ff8162e1700 1 -- 172.21.15.194:0/1373021124 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7ff7fc02ff60 msgr2=0x7ff7fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:38.321 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:38.322+0000 7ff8162e1700 1 --2- 172.21.15.194:0/1373021124 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7ff7fc02ff60 0x7ff7fc032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7ff7f80096a0 tx=0x7ff7f8006b40).stop 2024-01-18T06:56:38.321 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:38.322+0000 7ff8162e1700 1 -- 172.21.15.194:0/1373021124 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff8100f5410 msgr2=0x7ff8100fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:38.321 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:38.322+0000 7ff8162e1700 1 --2- 172.21.15.194:0/1373021124 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff8100f5410 0x7ff8100fd3a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7ff800013510 tx=0x7ff80000b040).stop 2024-01-18T06:56:38.322 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:38.322+0000 7ff8162e1700 1 -- 172.21.15.194:0/1373021124 shutdown_connections 2024-01-18T06:56:38.322 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:38.322+0000 7ff8162e1700 1 --2- 172.21.15.194:0/1373021124 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7ff7fc02ff60 0x7ff7fc032420 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:38.322 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:38.322+0000 7ff8162e1700 1 --2- 172.21.15.194:0/1373021124 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff8100f5410 0x7ff8100fd3a0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:38.322 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:38.322+0000 7ff8162e1700 1 -- 172.21.15.194:0/1373021124 >> 172.21.15.194:0/1373021124 conn(0x7ff8100f0b70 msgr2=0x7ff81005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:56:38.322 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:38.322+0000 7ff8162e1700 1 -- 172.21.15.194:0/1373021124 shutdown_connections 2024-01-18T06:56:38.322 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:38.322+0000 7ff8162e1700 1 -- 172.21.15.194:0/1373021124 wait complete. 2024-01-18T06:56:38.322 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:56:39.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:38 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:56:38.052499+0000 mgr.smithi047.fuhzzr (mgr.14162) 252 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:56:39.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:38 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:56:38.321098+0000 mon.smithi047 (mon.0) 305 : audit [DBG] from='client.? 172.21.15.194:0/1373021124' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:56:39.706 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:56:39.706 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:56:40.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:40.833+0000 7f50d439d700 1 -- 172.21.15.194:0/3144091894 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50cc0f5410 msgr2=0x7f50cc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:40.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:40.833+0000 7f50d439d700 1 --2- 172.21.15.194:0/3144091894 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50cc0f5410 0x7f50cc0f5830 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f50c8009fb0 tx=0x7f50c8009b30).stop 2024-01-18T06:56:40.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:40.833+0000 7f50d439d700 1 -- 172.21.15.194:0/3144091894 shutdown_connections 2024-01-18T06:56:40.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:40.833+0000 7f50d439d700 1 --2- 172.21.15.194:0/3144091894 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50cc0f5410 0x7f50cc0f5830 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:40.835 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:40.833+0000 7f50d439d700 1 -- 172.21.15.194:0/3144091894 >> 172.21.15.194:0/3144091894 conn(0x7f50cc0f0b70 msgr2=0x7f50cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:56:40.835 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:40.833+0000 7f50d439d700 1 -- 172.21.15.194:0/3144091894 shutdown_connections 2024-01-18T06:56:40.835 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:40.833+0000 7f50d439d700 1 -- 172.21.15.194:0/3144091894 wait complete. 2024-01-18T06:56:40.835 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:40.834+0000 7f50d439d700 1 Processor -- start 2024-01-18T06:56:40.835 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:40.834+0000 7f50d439d700 1 -- start start 2024-01-18T06:56:40.835 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:40.834+0000 7f50d439d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50cc0f5410 0x7f50cc0fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:56:40.835 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:40.834+0000 7f50d439d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50cc0fb820 con 0x7f50cc0f5410 2024-01-18T06:56:40.835 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:40.834+0000 7f50d2139700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50cc0f5410 0x7f50cc0fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:56:40.835 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:40.834+0000 7f50d2139700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50cc0f5410 0x7f50cc0fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:43256/0 (socket says 172.21.15.194:43256) 2024-01-18T06:56:40.835 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:40.834+0000 7f50d2139700 1 -- 172.21.15.194:0/1142074079 learned_addr learned my addr 172.21.15.194:0/1142074079 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:56:40.835 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:40.834+0000 7f50d2139700 1 -- 172.21.15.194:0/1142074079 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50c8005040 con 0x7f50cc0f5410 2024-01-18T06:56:40.835 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:40.834+0000 7f50d2139700 1 --2- 172.21.15.194:0/1142074079 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50cc0f5410 0x7f50cc0fd3a0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f50c8013510 tx=0x7f50c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:56:40.835 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:40.835+0000 7f50c2ffd700 1 -- 172.21.15.194:0/1142074079 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50c8009320 con 0x7f50cc0f5410 2024-01-18T06:56:40.836 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:40.835+0000 7f50d439d700 1 -- 172.21.15.194:0/1142074079 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50cc0fb9c0 con 0x7f50cc0f5410 2024-01-18T06:56:40.836 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:40.835+0000 7f50c2ffd700 1 -- 172.21.15.194:0/1142074079 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f50c8013f00 con 0x7f50cc0f5410 2024-01-18T06:56:40.836 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:40.835+0000 7f50d439d700 1 -- 172.21.15.194:0/1142074079 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50cc0fbe50 con 0x7f50cc0f5410 2024-01-18T06:56:40.836 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:40.835+0000 7f50c2ffd700 1 -- 172.21.15.194:0/1142074079 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50c8014d90 con 0x7f50cc0f5410 2024-01-18T06:56:40.837 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:40.835+0000 7f50c2ffd700 1 -- 172.21.15.194:0/1142074079 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f50c80095f0 con 0x7f50cc0f5410 2024-01-18T06:56:40.837 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:40.835+0000 7f50d439d700 1 -- 172.21.15.194:0/1142074079 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50cc059100 con 0x7f50cc0f5410 2024-01-18T06:56:40.837 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:40.836+0000 7f50c2ffd700 1 --2- 172.21.15.194:0/1142074079 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f50b802ff60 0x7f50b8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:56:40.837 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:40.836+0000 7f50c2ffd700 1 -- 172.21.15.194:0/1142074079 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f50c8040fd0 con 0x7f50cc0f5410 2024-01-18T06:56:40.837 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:40.836+0000 7f50d1938700 1 --2- 172.21.15.194:0/1142074079 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f50b802ff60 0x7f50b8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:56:40.837 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:40.836+0000 7f50d1938700 1 --2- 172.21.15.194:0/1142074079 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f50b802ff60 0x7f50b8032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f50bc0096a0 tx=0x7f50bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:56:40.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:40.840+0000 7f50c2ffd700 1 -- 172.21.15.194:0/1142074079 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f50c8014420 con 0x7f50cc0f5410 2024-01-18T06:56:41.045 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:41.046+0000 7f50d439d700 1 -- 172.21.15.194:0/1142074079 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f50cc045ad0 con 0x7f50cc0f5410 2024-01-18T06:56:41.046 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:41.047+0000 7f50c2ffd700 1 -- 172.21.15.194:0/1142074079 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f50c8010230 con 0x7f50cc0f5410 2024-01-18T06:56:41.046 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:56:41.046 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:56:41.048 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:41.048+0000 7f50d439d700 1 -- 172.21.15.194:0/1142074079 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f50b802ff60 msgr2=0x7f50b8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:41.048 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:41.048+0000 7f50d439d700 1 --2- 172.21.15.194:0/1142074079 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f50b802ff60 0x7f50b8032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f50bc0096a0 tx=0x7f50bc006b40).stop 2024-01-18T06:56:41.048 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:41.048+0000 7f50d439d700 1 -- 172.21.15.194:0/1142074079 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50cc0f5410 msgr2=0x7f50cc0fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:41.048 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:41.048+0000 7f50d439d700 1 --2- 172.21.15.194:0/1142074079 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50cc0f5410 0x7f50cc0fd3a0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f50c8013510 tx=0x7f50c800b040).stop 2024-01-18T06:56:41.048 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:41.048+0000 7f50d439d700 1 -- 172.21.15.194:0/1142074079 shutdown_connections 2024-01-18T06:56:41.048 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:41.048+0000 7f50d439d700 1 --2- 172.21.15.194:0/1142074079 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f50b802ff60 0x7f50b8032420 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:41.048 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:41.048+0000 7f50d439d700 1 --2- 172.21.15.194:0/1142074079 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50cc0f5410 0x7f50cc0fd3a0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:41.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:41.048+0000 7f50d439d700 1 -- 172.21.15.194:0/1142074079 >> 172.21.15.194:0/1142074079 conn(0x7f50cc0f0b70 msgr2=0x7f50cc05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:56:41.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:41.048+0000 7f50d439d700 1 -- 172.21.15.194:0/1142074079 shutdown_connections 2024-01-18T06:56:41.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:41.048+0000 7f50d439d700 1 -- 172.21.15.194:0/1142074079 wait complete. 2024-01-18T06:56:41.049 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:56:41.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:41 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:56:40.052895+0000 mgr.smithi047.fuhzzr (mgr.14162) 253 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:56:41.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:41 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:56:41.047834+0000 mon.smithi047 (mon.0) 306 : audit [DBG] from='client.? 172.21.15.194:0/1142074079' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:56:42.439 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:56:42.439 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:56:43.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:43 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:56:42.053224+0000 mgr.smithi047.fuhzzr (mgr.14162) 254 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:56:43.655 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:43.655+0000 7f3af0f79700 1 -- 172.21.15.194:0/2839289918 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3aec0f5410 msgr2=0x7f3aec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:43.655 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:43.655+0000 7f3af0f79700 1 --2- 172.21.15.194:0/2839289918 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3aec0f5410 0x7f3aec0f5830 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f3ae0009fb0 tx=0x7f3ae0009b30).stop 2024-01-18T06:56:43.655 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:43.655+0000 7f3af0f79700 1 -- 172.21.15.194:0/2839289918 shutdown_connections 2024-01-18T06:56:43.655 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:43.655+0000 7f3af0f79700 1 --2- 172.21.15.194:0/2839289918 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3aec0f5410 0x7f3aec0f5830 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:43.655 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:43.655+0000 7f3af0f79700 1 -- 172.21.15.194:0/2839289918 >> 172.21.15.194:0/2839289918 conn(0x7f3aec0f0b70 msgr2=0x7f3aec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:56:43.655 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:43.655+0000 7f3af0f79700 1 -- 172.21.15.194:0/2839289918 shutdown_connections 2024-01-18T06:56:43.655 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:43.655+0000 7f3af0f79700 1 -- 172.21.15.194:0/2839289918 wait complete. 2024-01-18T06:56:43.656 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:43.656+0000 7f3af0f79700 1 Processor -- start 2024-01-18T06:56:43.656 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:43.656+0000 7f3af0f79700 1 -- start start 2024-01-18T06:56:43.657 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:43.656+0000 7f3af0f79700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3aec0f5410 0x7f3aec105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:56:43.657 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:43.656+0000 7f3af0f79700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aec105b20 con 0x7f3aec0f5410 2024-01-18T06:56:43.657 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:43.656+0000 7f3aea59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3aec0f5410 0x7f3aec105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:56:43.657 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:43.656+0000 7f3aea59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3aec0f5410 0x7f3aec105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:43266/0 (socket says 172.21.15.194:43266) 2024-01-18T06:56:43.658 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:43.656+0000 7f3aea59c700 1 -- 172.21.15.194:0/1539799384 learned_addr learned my addr 172.21.15.194:0/1539799384 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:56:43.658 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:43.657+0000 7f3aea59c700 1 -- 172.21.15.194:0/1539799384 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae0005040 con 0x7f3aec0f5410 2024-01-18T06:56:43.658 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:43.657+0000 7f3aea59c700 1 --2- 172.21.15.194:0/1539799384 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3aec0f5410 0x7f3aec105630 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f3ae0013510 tx=0x7f3ae000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:56:43.658 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:43.657+0000 7f3adaffd700 1 -- 172.21.15.194:0/1539799384 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ae0009320 con 0x7f3aec0f5410 2024-01-18T06:56:43.658 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:43.657+0000 7f3af0f79700 1 -- 172.21.15.194:0/1539799384 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3aec105d20 con 0x7f3aec0f5410 2024-01-18T06:56:43.658 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:43.657+0000 7f3adaffd700 1 -- 172.21.15.194:0/1539799384 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3ae0013f00 con 0x7f3aec0f5410 2024-01-18T06:56:43.658 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:43.657+0000 7f3adaffd700 1 -- 172.21.15.194:0/1539799384 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ae0014d90 con 0x7f3aec0f5410 2024-01-18T06:56:43.658 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:43.657+0000 7f3af0f79700 1 -- 172.21.15.194:0/1539799384 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3aec1061b0 con 0x7f3aec0f5410 2024-01-18T06:56:43.658 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:43.658+0000 7f3adaffd700 1 -- 172.21.15.194:0/1539799384 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3ae00095f0 con 0x7f3aec0f5410 2024-01-18T06:56:43.658 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:43.658+0000 7f3af0f79700 1 -- 172.21.15.194:0/1539799384 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3aec045ad0 con 0x7f3aec0f5410 2024-01-18T06:56:43.658 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:43.658+0000 7f3adaffd700 1 --2- 172.21.15.194:0/1539799384 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3ad402ff60 0x7f3ad4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:56:43.658 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:43.658+0000 7f3adaffd700 1 -- 172.21.15.194:0/1539799384 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3ae00410c0 con 0x7f3aec0f5410 2024-01-18T06:56:43.658 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:43.658+0000 7f3ae9d9b700 1 --2- 172.21.15.194:0/1539799384 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3ad402ff60 0x7f3ad4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:56:43.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:43.659+0000 7f3ae9d9b700 1 --2- 172.21.15.194:0/1539799384 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3ad402ff60 0x7f3ad4032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f3adc0096a0 tx=0x7f3adc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:56:43.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:43.663+0000 7f3adaffd700 1 -- 172.21.15.194:0/1539799384 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3ae0014420 con 0x7f3aec0f5410 2024-01-18T06:56:43.867 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:43.867+0000 7f3af0f79700 1 -- 172.21.15.194:0/1539799384 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3aec0fae60 con 0x7f3aec0f5410 2024-01-18T06:56:43.867 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:43.868+0000 7f3adaffd700 1 -- 172.21.15.194:0/1539799384 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3ae0010230 con 0x7f3aec0f5410 2024-01-18T06:56:43.867 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:56:43.868 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:56:43.869 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:43.869+0000 7f3af0f79700 1 -- 172.21.15.194:0/1539799384 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3ad402ff60 msgr2=0x7f3ad4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:43.869 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:43.869+0000 7f3af0f79700 1 --2- 172.21.15.194:0/1539799384 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3ad402ff60 0x7f3ad4032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f3adc0096a0 tx=0x7f3adc006b40).stop 2024-01-18T06:56:43.869 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:43.869+0000 7f3af0f79700 1 -- 172.21.15.194:0/1539799384 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3aec0f5410 msgr2=0x7f3aec105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:43.869 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:43.869+0000 7f3af0f79700 1 --2- 172.21.15.194:0/1539799384 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3aec0f5410 0x7f3aec105630 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f3ae0013510 tx=0x7f3ae000b040).stop 2024-01-18T06:56:43.869 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:43.869+0000 7f3af0f79700 1 -- 172.21.15.194:0/1539799384 shutdown_connections 2024-01-18T06:56:43.869 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:43.869+0000 7f3af0f79700 1 --2- 172.21.15.194:0/1539799384 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f3ad402ff60 0x7f3ad4032420 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:43.869 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:43.869+0000 7f3af0f79700 1 --2- 172.21.15.194:0/1539799384 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3aec0f5410 0x7f3aec105630 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:43.869 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:43.869+0000 7f3af0f79700 1 -- 172.21.15.194:0/1539799384 >> 172.21.15.194:0/1539799384 conn(0x7f3aec0f0b70 msgr2=0x7f3aec0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:56:43.869 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:43.869+0000 7f3af0f79700 1 -- 172.21.15.194:0/1539799384 shutdown_connections 2024-01-18T06:56:43.870 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:43.869+0000 7f3af0f79700 1 -- 172.21.15.194:0/1539799384 wait complete. 2024-01-18T06:56:43.870 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:56:44.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:44 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:56:43.868900+0000 mon.smithi047 (mon.0) 307 : audit [DBG] from='client.? 172.21.15.194:0/1539799384' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:56:45.244 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:56:45.245 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:56:45.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:45 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:56:44.053576+0000 mgr.smithi047.fuhzzr (mgr.14162) 255 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:56:46.393 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:46.393+0000 7fbfcb59e700 1 -- 172.21.15.194:0/1192567664 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbfcc0f51a0 msgr2=0x7fbfcc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:46.393 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:46.393+0000 7fbfcb59e700 1 --2- 172.21.15.194:0/1192567664 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbfcc0f51a0 0x7fbfcc0f55c0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7fbfbc009fb0 tx=0x7fbfbc009b30).stop 2024-01-18T06:56:46.393 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:46.393+0000 7fbfcb59e700 1 -- 172.21.15.194:0/1192567664 shutdown_connections 2024-01-18T06:56:46.393 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:46.393+0000 7fbfcb59e700 1 --2- 172.21.15.194:0/1192567664 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbfcc0f51a0 0x7fbfcc0f55c0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:46.393 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:46.393+0000 7fbfcb59e700 1 -- 172.21.15.194:0/1192567664 >> 172.21.15.194:0/1192567664 conn(0x7fbfcc0f09a0 msgr2=0x7fbfcc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:56:46.393 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:46.393+0000 7fbfcb59e700 1 -- 172.21.15.194:0/1192567664 shutdown_connections 2024-01-18T06:56:46.394 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:46.393+0000 7fbfcb59e700 1 -- 172.21.15.194:0/1192567664 wait complete. 2024-01-18T06:56:46.394 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:46.393+0000 7fbfcb59e700 1 Processor -- start 2024-01-18T06:56:46.394 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:46.393+0000 7fbfcb59e700 1 -- start start 2024-01-18T06:56:46.394 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:46.393+0000 7fbfcb59e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbfcc0f51a0 0x7fbfcc1091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:56:46.394 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:46.393+0000 7fbfcb59e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfcc060240 con 0x7fbfcc0f51a0 2024-01-18T06:56:46.394 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:46.394+0000 7fbfca59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbfcc0f51a0 0x7fbfcc1091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:56:46.394 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:46.394+0000 7fbfca59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbfcc0f51a0 0x7fbfcc1091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:43272/0 (socket says 172.21.15.194:43272) 2024-01-18T06:56:46.394 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:46.394+0000 7fbfca59c700 1 -- 172.21.15.194:0/1041287957 learned_addr learned my addr 172.21.15.194:0/1041287957 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:56:46.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:46.394+0000 7fbfca59c700 1 -- 172.21.15.194:0/1041287957 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfbc005040 con 0x7fbfcc0f51a0 2024-01-18T06:56:46.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:46.394+0000 7fbfca59c700 1 --2- 172.21.15.194:0/1041287957 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbfcc0f51a0 0x7fbfcc1091b0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7fbfbc013510 tx=0x7fbfbc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:56:46.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:46.394+0000 7fbfbaffd700 1 -- 172.21.15.194:0/1041287957 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfbc009320 con 0x7fbfcc0f51a0 2024-01-18T06:56:46.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:46.394+0000 7fbfcb59e700 1 -- 172.21.15.194:0/1041287957 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfcc060440 con 0x7fbfcc0f51a0 2024-01-18T06:56:46.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:46.394+0000 7fbfcb59e700 1 -- 172.21.15.194:0/1041287957 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfcc109860 con 0x7fbfcc0f51a0 2024-01-18T06:56:46.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:46.394+0000 7fbfbaffd700 1 -- 172.21.15.194:0/1041287957 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbfbc013f00 con 0x7fbfcc0f51a0 2024-01-18T06:56:46.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:46.394+0000 7fbfbaffd700 1 -- 172.21.15.194:0/1041287957 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfbc014d90 con 0x7fbfcc0f51a0 2024-01-18T06:56:46.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:46.395+0000 7fbfbaffd700 1 -- 172.21.15.194:0/1041287957 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbfbc0263a0 con 0x7fbfcc0f51a0 2024-01-18T06:56:46.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:46.395+0000 7fbfcb59e700 1 -- 172.21.15.194:0/1041287957 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfb0004fa0 con 0x7fbfcc0f51a0 2024-01-18T06:56:46.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:46.395+0000 7fbfbaffd700 1 --2- 172.21.15.194:0/1041287957 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fbfb402ff60 0x7fbfb4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:56:46.397 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:46.395+0000 7fbfbaffd700 1 -- 172.21.15.194:0/1041287957 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbfbc041260 con 0x7fbfcc0f51a0 2024-01-18T06:56:46.397 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:46.396+0000 7fbfc9d9b700 1 --2- 172.21.15.194:0/1041287957 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fbfb402ff60 0x7fbfb4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:56:46.397 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:46.396+0000 7fbfc9d9b700 1 --2- 172.21.15.194:0/1041287957 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fbfb402ff60 0x7fbfb4032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fbfc00096a0 tx=0x7fbfc0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:56:46.399 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:46.400+0000 7fbfbaffd700 1 -- 172.21.15.194:0/1041287957 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbfbc012510 con 0x7fbfcc0f51a0 2024-01-18T06:56:46.606 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:46.606+0000 7fbfcb59e700 1 -- 172.21.15.194:0/1041287957 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbfb0005670 con 0x7fbfcc0f51a0 2024-01-18T06:56:46.606 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:46.607+0000 7fbfbaffd700 1 -- 172.21.15.194:0/1041287957 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbfbc012330 con 0x7fbfcc0f51a0 2024-01-18T06:56:46.607 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:56:46.607 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:56:46.608 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:46.608+0000 7fbfcb59e700 1 -- 172.21.15.194:0/1041287957 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fbfb402ff60 msgr2=0x7fbfb4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:46.608 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:46.608+0000 7fbfcb59e700 1 --2- 172.21.15.194:0/1041287957 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fbfb402ff60 0x7fbfb4032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fbfc00096a0 tx=0x7fbfc0006b40).stop 2024-01-18T06:56:46.608 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:46.608+0000 7fbfcb59e700 1 -- 172.21.15.194:0/1041287957 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbfcc0f51a0 msgr2=0x7fbfcc1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:46.608 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:46.608+0000 7fbfcb59e700 1 --2- 172.21.15.194:0/1041287957 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbfcc0f51a0 0x7fbfcc1091b0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7fbfbc013510 tx=0x7fbfbc00b040).stop 2024-01-18T06:56:46.608 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:46.608+0000 7fbfcb59e700 1 -- 172.21.15.194:0/1041287957 shutdown_connections 2024-01-18T06:56:46.608 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:46.608+0000 7fbfcb59e700 1 --2- 172.21.15.194:0/1041287957 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fbfb402ff60 0x7fbfb4032420 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:46.608 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:46.608+0000 7fbfcb59e700 1 --2- 172.21.15.194:0/1041287957 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbfcc0f51a0 0x7fbfcc1091b0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:46.608 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:46.608+0000 7fbfcb59e700 1 -- 172.21.15.194:0/1041287957 >> 172.21.15.194:0/1041287957 conn(0x7fbfcc0f09a0 msgr2=0x7fbfcc0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:56:46.608 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:46.608+0000 7fbfcb59e700 1 -- 172.21.15.194:0/1041287957 shutdown_connections 2024-01-18T06:56:46.608 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:46.608+0000 7fbfcb59e700 1 -- 172.21.15.194:0/1041287957 wait complete. 2024-01-18T06:56:46.609 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:56:47.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:47 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:56:46.053886+0000 mgr.smithi047.fuhzzr (mgr.14162) 256 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:56:47.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:47 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:56:46.607781+0000 mon.smithi047 (mon.0) 308 : audit [DBG] from='client.? 172.21.15.194:0/1041287957' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:56:47.999 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:56:48.000 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:56:49.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:49.200+0000 7fca6e7a4700 1 -- 172.21.15.194:0/235966954 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca68060280 msgr2=0x7fca680f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:49.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:49.200+0000 7fca6e7a4700 1 --2- 172.21.15.194:0/235966954 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca68060280 0x7fca680f90f0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7fca58009fb0 tx=0x7fca58009b30).stop 2024-01-18T06:56:49.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:49.200+0000 7fca6e7a4700 1 -- 172.21.15.194:0/235966954 shutdown_connections 2024-01-18T06:56:49.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:49.200+0000 7fca6e7a4700 1 --2- 172.21.15.194:0/235966954 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca68060280 0x7fca680f90f0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:49.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:49.200+0000 7fca6e7a4700 1 -- 172.21.15.194:0/235966954 >> 172.21.15.194:0/235966954 conn(0x7fca680f0b70 msgr2=0x7fca680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:56:49.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:49.201+0000 7fca6e7a4700 1 -- 172.21.15.194:0/235966954 shutdown_connections 2024-01-18T06:56:49.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:49.201+0000 7fca6e7a4700 1 -- 172.21.15.194:0/235966954 wait complete. 2024-01-18T06:56:49.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:49.201+0000 7fca6e7a4700 1 Processor -- start 2024-01-18T06:56:49.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:49.201+0000 7fca6e7a4700 1 -- start start 2024-01-18T06:56:49.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:49.201+0000 7fca6e7a4700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca68060280 0x7fca681066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:56:49.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:49.201+0000 7fca6e7a4700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca68106bd0 con 0x7fca68060280 2024-01-18T06:56:49.202 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:49.201+0000 7fca67fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca68060280 0x7fca681066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:56:49.202 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:49.201+0000 7fca67fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca68060280 0x7fca681066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:53248/0 (socket says 172.21.15.194:53248) 2024-01-18T06:56:49.202 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:49.201+0000 7fca67fff700 1 -- 172.21.15.194:0/3230035926 learned_addr learned my addr 172.21.15.194:0/3230035926 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:56:49.203 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:49.202+0000 7fca67fff700 1 -- 172.21.15.194:0/3230035926 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca58005040 con 0x7fca68060280 2024-01-18T06:56:49.203 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:49.202+0000 7fca67fff700 1 --2- 172.21.15.194:0/3230035926 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca68060280 0x7fca681066e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7fca58013510 tx=0x7fca5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:56:49.203 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:49.202+0000 7fca657fa700 1 -- 172.21.15.194:0/3230035926 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca58009320 con 0x7fca68060280 2024-01-18T06:56:49.203 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:49.202+0000 7fca6e7a4700 1 -- 172.21.15.194:0/3230035926 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca68106dd0 con 0x7fca68060280 2024-01-18T06:56:49.203 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:49.202+0000 7fca657fa700 1 -- 172.21.15.194:0/3230035926 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fca58013f00 con 0x7fca68060280 2024-01-18T06:56:49.203 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:49.202+0000 7fca657fa700 1 -- 172.21.15.194:0/3230035926 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca58014dc0 con 0x7fca68060280 2024-01-18T06:56:49.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:49.202+0000 7fca6e7a4700 1 -- 172.21.15.194:0/3230035926 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca68107260 con 0x7fca68060280 2024-01-18T06:56:49.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:49.203+0000 7fca657fa700 1 -- 172.21.15.194:0/3230035926 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fca5801d3d0 con 0x7fca68060280 2024-01-18T06:56:49.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:49.203+0000 7fca657fa700 1 --2- 172.21.15.194:0/3230035926 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fca5402ff60 0x7fca54032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:56:49.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:49.203+0000 7fca657fa700 1 -- 172.21.15.194:0/3230035926 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fca58041120 con 0x7fca68060280 2024-01-18T06:56:49.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:49.203+0000 7fca6e7a4700 1 -- 172.21.15.194:0/3230035926 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca68045ad0 con 0x7fca68060280 2024-01-18T06:56:49.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:49.203+0000 7fca5ffff700 1 --2- 172.21.15.194:0/3230035926 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fca5402ff60 0x7fca54032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:56:49.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:49.204+0000 7fca5ffff700 1 --2- 172.21.15.194:0/3230035926 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fca5402ff60 0x7fca54032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fca500096a0 tx=0x7fca50006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:56:49.207 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:49.208+0000 7fca657fa700 1 -- 172.21.15.194:0/3230035926 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fca58012920 con 0x7fca68060280 2024-01-18T06:56:49.412 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:49.413+0000 7fca6e7a4700 1 -- 172.21.15.194:0/3230035926 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fca68107f80 con 0x7fca68060280 2024-01-18T06:56:49.413 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:49.414+0000 7fca657fa700 1 -- 172.21.15.194:0/3230035926 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fca5802b020 con 0x7fca68060280 2024-01-18T06:56:49.413 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:56:49.413 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:56:49.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:49.415+0000 7fca6e7a4700 1 -- 172.21.15.194:0/3230035926 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fca5402ff60 msgr2=0x7fca54032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:49.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:49.415+0000 7fca6e7a4700 1 --2- 172.21.15.194:0/3230035926 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fca5402ff60 0x7fca54032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fca500096a0 tx=0x7fca50006b40).stop 2024-01-18T06:56:49.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:49.415+0000 7fca6e7a4700 1 -- 172.21.15.194:0/3230035926 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca68060280 msgr2=0x7fca681066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:49.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:49.415+0000 7fca6e7a4700 1 --2- 172.21.15.194:0/3230035926 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca68060280 0x7fca681066e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7fca58013510 tx=0x7fca5800b040).stop 2024-01-18T06:56:49.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:49.415+0000 7fca6e7a4700 1 -- 172.21.15.194:0/3230035926 shutdown_connections 2024-01-18T06:56:49.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:49.415+0000 7fca6e7a4700 1 --2- 172.21.15.194:0/3230035926 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fca5402ff60 0x7fca54032420 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:49.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:49.415+0000 7fca6e7a4700 1 --2- 172.21.15.194:0/3230035926 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca68060280 0x7fca681066e0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:49.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:49.415+0000 7fca6e7a4700 1 -- 172.21.15.194:0/3230035926 >> 172.21.15.194:0/3230035926 conn(0x7fca680f0b70 msgr2=0x7fca680f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:56:49.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:49.415+0000 7fca6e7a4700 1 -- 172.21.15.194:0/3230035926 shutdown_connections 2024-01-18T06:56:49.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:49.415+0000 7fca6e7a4700 1 -- 172.21.15.194:0/3230035926 wait complete. 2024-01-18T06:56:49.416 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:56:49.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:48 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:56:48.054215+0000 mgr.smithi047.fuhzzr (mgr.14162) 257 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:56:50.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:49 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:56:49.414641+0000 mon.smithi047 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.194:0/3230035926' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:56:50.798 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:56:50.799 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:56:51.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:50 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:56:50.054603+0000 mgr.smithi047.fuhzzr (mgr.14162) 258 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:56:51.996 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:51.996+0000 7fb8fa8c1700 1 -- 172.21.15.194:0/487047623 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8f40f6510 msgr2=0x7fb8f40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:51.996 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:51.996+0000 7fb8fa8c1700 1 --2- 172.21.15.194:0/487047623 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8f40f6510 0x7fb8f40f6930 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7fb8e4009fb0 tx=0x7fb8e4009b30).stop 2024-01-18T06:56:51.996 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:51.996+0000 7fb8fa8c1700 1 -- 172.21.15.194:0/487047623 shutdown_connections 2024-01-18T06:56:51.996 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:51.996+0000 7fb8fa8c1700 1 --2- 172.21.15.194:0/487047623 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8f40f6510 0x7fb8f40f6930 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:51.996 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:51.996+0000 7fb8fa8c1700 1 -- 172.21.15.194:0/487047623 >> 172.21.15.194:0/487047623 conn(0x7fb8f40f1c70 msgr2=0x7fb8f40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:56:51.996 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:51.996+0000 7fb8fa8c1700 1 -- 172.21.15.194:0/487047623 shutdown_connections 2024-01-18T06:56:51.996 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:51.996+0000 7fb8fa8c1700 1 -- 172.21.15.194:0/487047623 wait complete. 2024-01-18T06:56:51.997 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:51.996+0000 7fb8fa8c1700 1 Processor -- start 2024-01-18T06:56:51.997 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:51.997+0000 7fb8fa8c1700 1 -- start start 2024-01-18T06:56:51.997 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:51.997+0000 7fb8fa8c1700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8f40f6510 0x7fb8f4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:56:51.997 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:51.997+0000 7fb8fa8c1700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f4106c20 con 0x7fb8f40f6510 2024-01-18T06:56:51.998 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:51.997+0000 7fb8f3fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8f40f6510 0x7fb8f4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:56:51.999 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:51.997+0000 7fb8f3fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8f40f6510 0x7fb8f4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:53262/0 (socket says 172.21.15.194:53262) 2024-01-18T06:56:51.999 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:51.997+0000 7fb8f3fff700 1 -- 172.21.15.194:0/3318263514 learned_addr learned my addr 172.21.15.194:0/3318263514 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:56:51.999 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:51.997+0000 7fb8f3fff700 1 -- 172.21.15.194:0/3318263514 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e4005040 con 0x7fb8f40f6510 2024-01-18T06:56:51.999 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:51.998+0000 7fb8f3fff700 1 --2- 172.21.15.194:0/3318263514 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8f40f6510 0x7fb8f4106730 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fb8e4013510 tx=0x7fb8e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:56:51.999 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:51.998+0000 7fb8f0ff9700 1 -- 172.21.15.194:0/3318263514 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8e4009320 con 0x7fb8f40f6510 2024-01-18T06:56:51.999 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:51.998+0000 7fb8fa8c1700 1 -- 172.21.15.194:0/3318263514 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8f4106e20 con 0x7fb8f40f6510 2024-01-18T06:56:51.999 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:51.998+0000 7fb8f0ff9700 1 -- 172.21.15.194:0/3318263514 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb8e4013f00 con 0x7fb8f40f6510 2024-01-18T06:56:51.999 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:51.998+0000 7fb8fa8c1700 1 -- 172.21.15.194:0/3318263514 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8f41072b0 con 0x7fb8f40f6510 2024-01-18T06:56:51.999 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:51.998+0000 7fb8f0ff9700 1 -- 172.21.15.194:0/3318263514 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8e4014d90 con 0x7fb8f40f6510 2024-01-18T06:56:51.999 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:51.999+0000 7fb8fa8c1700 1 -- 172.21.15.194:0/3318263514 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8f4045ad0 con 0x7fb8f40f6510 2024-01-18T06:56:51.999 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:51.999+0000 7fb8f0ff9700 1 -- 172.21.15.194:0/3318263514 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb8e40095f0 con 0x7fb8f40f6510 2024-01-18T06:56:51.999 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:51.999+0000 7fb8f0ff9700 1 --2- 172.21.15.194:0/3318263514 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb8e002ff60 0x7fb8e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:56:51.999 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:51.999+0000 7fb8f0ff9700 1 -- 172.21.15.194:0/3318263514 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb8e40410c0 con 0x7fb8f40f6510 2024-01-18T06:56:52.000 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:51.999+0000 7fb8f37fe700 1 --2- 172.21.15.194:0/3318263514 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb8e002ff60 0x7fb8e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:56:52.000 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:51.999+0000 7fb8f37fe700 1 --2- 172.21.15.194:0/3318263514 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb8e002ff60 0x7fb8e0032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fb8dc0096a0 tx=0x7fb8dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:56:52.003 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:52.003+0000 7fb8f0ff9700 1 -- 172.21.15.194:0/3318263514 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb8e4014420 con 0x7fb8f40f6510 2024-01-18T06:56:52.208 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:52.208+0000 7fb8fa8c1700 1 -- 172.21.15.194:0/3318263514 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb8f40fbf60 con 0x7fb8f40f6510 2024-01-18T06:56:52.208 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:52.209+0000 7fb8f0ff9700 1 -- 172.21.15.194:0/3318263514 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb8e4010230 con 0x7fb8f40f6510 2024-01-18T06:56:52.208 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:56:52.208 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:56:52.210 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:52.210+0000 7fb8fa8c1700 1 -- 172.21.15.194:0/3318263514 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb8e002ff60 msgr2=0x7fb8e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:52.210 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:52.210+0000 7fb8fa8c1700 1 --2- 172.21.15.194:0/3318263514 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb8e002ff60 0x7fb8e0032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fb8dc0096a0 tx=0x7fb8dc006b40).stop 2024-01-18T06:56:52.210 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:52.210+0000 7fb8fa8c1700 1 -- 172.21.15.194:0/3318263514 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8f40f6510 msgr2=0x7fb8f4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:52.210 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:52.210+0000 7fb8fa8c1700 1 --2- 172.21.15.194:0/3318263514 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8f40f6510 0x7fb8f4106730 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fb8e4013510 tx=0x7fb8e400b040).stop 2024-01-18T06:56:52.210 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:52.210+0000 7fb8fa8c1700 1 -- 172.21.15.194:0/3318263514 shutdown_connections 2024-01-18T06:56:52.210 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:52.210+0000 7fb8fa8c1700 1 --2- 172.21.15.194:0/3318263514 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb8e002ff60 0x7fb8e0032420 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:52.210 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:52.210+0000 7fb8fa8c1700 1 --2- 172.21.15.194:0/3318263514 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8f40f6510 0x7fb8f4106730 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:52.210 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:52.210+0000 7fb8fa8c1700 1 -- 172.21.15.194:0/3318263514 >> 172.21.15.194:0/3318263514 conn(0x7fb8f40f1c70 msgr2=0x7fb8f40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:56:52.211 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:52.211+0000 7fb8fa8c1700 1 -- 172.21.15.194:0/3318263514 shutdown_connections 2024-01-18T06:56:52.211 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:52.211+0000 7fb8fa8c1700 1 -- 172.21.15.194:0/3318263514 wait complete. 2024-01-18T06:56:52.211 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:56:53.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:53 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:56:52.054987+0000 mgr.smithi047.fuhzzr (mgr.14162) 259 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:56:53.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:53 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:56:52.209821+0000 mon.smithi047 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.194:0/3318263514' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:56:53.586 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:56:53.586 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:56:54.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:54.737+0000 7fa4b160d700 1 -- 172.21.15.194:0/4197802049 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4ac0601e0 msgr2=0x7fa4ac0f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:54.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:54.737+0000 7fa4b160d700 1 --2- 172.21.15.194:0/4197802049 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4ac0601e0 0x7fa4ac0f8f20 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7fa49c009fb0 tx=0x7fa49c009b30).stop 2024-01-18T06:56:54.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:54.737+0000 7fa4b160d700 1 -- 172.21.15.194:0/4197802049 shutdown_connections 2024-01-18T06:56:54.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:54.737+0000 7fa4b160d700 1 --2- 172.21.15.194:0/4197802049 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4ac0601e0 0x7fa4ac0f8f20 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:54.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:54.737+0000 7fa4b160d700 1 -- 172.21.15.194:0/4197802049 >> 172.21.15.194:0/4197802049 conn(0x7fa4ac0f09a0 msgr2=0x7fa4ac0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:56:54.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:54.737+0000 7fa4b160d700 1 -- 172.21.15.194:0/4197802049 shutdown_connections 2024-01-18T06:56:54.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:54.737+0000 7fa4b160d700 1 -- 172.21.15.194:0/4197802049 wait complete. 2024-01-18T06:56:54.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:54.738+0000 7fa4b160d700 1 Processor -- start 2024-01-18T06:56:54.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:54.738+0000 7fa4b160d700 1 -- start start 2024-01-18T06:56:54.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:54.738+0000 7fa4b160d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4ac0601e0 0x7fa4ac106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:56:54.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:54.738+0000 7fa4b160d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac1069f0 con 0x7fa4ac0601e0 2024-01-18T06:56:54.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:54.738+0000 7fa4abfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4ac0601e0 0x7fa4ac106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:56:54.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:54.738+0000 7fa4abfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4ac0601e0 0x7fa4ac106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:53286/0 (socket says 172.21.15.194:53286) 2024-01-18T06:56:54.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:54.738+0000 7fa4abfff700 1 -- 172.21.15.194:0/407082242 learned_addr learned my addr 172.21.15.194:0/407082242 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:56:54.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:54.739+0000 7fa4abfff700 1 -- 172.21.15.194:0/407082242 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa49c005040 con 0x7fa4ac0601e0 2024-01-18T06:56:54.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:54.739+0000 7fa4abfff700 1 --2- 172.21.15.194:0/407082242 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4ac0601e0 0x7fa4ac106500 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7fa49c013510 tx=0x7fa49c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:56:54.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:54.739+0000 7fa4a8ff9700 1 -- 172.21.15.194:0/407082242 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa49c009320 con 0x7fa4ac0601e0 2024-01-18T06:56:54.742 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:54.739+0000 7fa4b160d700 1 -- 172.21.15.194:0/407082242 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4ac106bf0 con 0x7fa4ac0601e0 2024-01-18T06:56:54.742 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:54.739+0000 7fa4a8ff9700 1 -- 172.21.15.194:0/407082242 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa49c013f00 con 0x7fa4ac0601e0 2024-01-18T06:56:54.742 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:54.739+0000 7fa4a8ff9700 1 -- 172.21.15.194:0/407082242 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa49c014d90 con 0x7fa4ac0601e0 2024-01-18T06:56:54.742 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:54.739+0000 7fa4b160d700 1 -- 172.21.15.194:0/407082242 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4ac107080 con 0x7fa4ac0601e0 2024-01-18T06:56:54.742 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:54.740+0000 7fa4a8ff9700 1 -- 172.21.15.194:0/407082242 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa49c014420 con 0x7fa4ac0601e0 2024-01-18T06:56:54.742 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:54.740+0000 7fa4b160d700 1 -- 172.21.15.194:0/407082242 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4ac045ad0 con 0x7fa4ac0601e0 2024-01-18T06:56:54.742 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:54.740+0000 7fa4a8ff9700 1 --2- 172.21.15.194:0/407082242 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa49402ffb0 0x7fa494032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:56:54.742 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:54.740+0000 7fa4a8ff9700 1 -- 172.21.15.194:0/407082242 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa49c041090 con 0x7fa4ac0601e0 2024-01-18T06:56:54.742 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:54.740+0000 7fa4ab7fe700 1 --2- 172.21.15.194:0/407082242 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa49402ffb0 0x7fa494032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:56:54.742 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:54.741+0000 7fa4ab7fe700 1 --2- 172.21.15.194:0/407082242 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa49402ffb0 0x7fa494032470 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fa4a00096a0 tx=0x7fa4a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:56:54.744 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:54.745+0000 7fa4a8ff9700 1 -- 172.21.15.194:0/407082242 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa49c06d2b0 con 0x7fa4ac0601e0 2024-01-18T06:56:54.950 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:54.951+0000 7fa4b160d700 1 -- 172.21.15.194:0/407082242 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa4ac107f30 con 0x7fa4ac0601e0 2024-01-18T06:56:54.951 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:54.952+0000 7fa4a8ff9700 1 -- 172.21.15.194:0/407082242 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa49c0095f0 con 0x7fa4ac0601e0 2024-01-18T06:56:54.951 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:56:54.951 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:56:54.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:54.953+0000 7fa4b160d700 1 -- 172.21.15.194:0/407082242 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa49402ffb0 msgr2=0x7fa494032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:54.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:54.953+0000 7fa4b160d700 1 --2- 172.21.15.194:0/407082242 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa49402ffb0 0x7fa494032470 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fa4a00096a0 tx=0x7fa4a0006b40).stop 2024-01-18T06:56:54.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:54.953+0000 7fa4b160d700 1 -- 172.21.15.194:0/407082242 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4ac0601e0 msgr2=0x7fa4ac106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:54.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:54.953+0000 7fa4b160d700 1 --2- 172.21.15.194:0/407082242 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4ac0601e0 0x7fa4ac106500 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7fa49c013510 tx=0x7fa49c00b040).stop 2024-01-18T06:56:54.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:54.953+0000 7fa4b160d700 1 -- 172.21.15.194:0/407082242 shutdown_connections 2024-01-18T06:56:54.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:54.953+0000 7fa4b160d700 1 --2- 172.21.15.194:0/407082242 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fa49402ffb0 0x7fa494032470 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:54.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:54.953+0000 7fa4b160d700 1 --2- 172.21.15.194:0/407082242 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4ac0601e0 0x7fa4ac106500 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:54.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:54.953+0000 7fa4b160d700 1 -- 172.21.15.194:0/407082242 >> 172.21.15.194:0/407082242 conn(0x7fa4ac0f09a0 msgr2=0x7fa4ac0f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:56:54.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:54.953+0000 7fa4b160d700 1 -- 172.21.15.194:0/407082242 shutdown_connections 2024-01-18T06:56:54.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:54.953+0000 7fa4b160d700 1 -- 172.21.15.194:0/407082242 wait complete. 2024-01-18T06:56:54.954 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:56:55.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:55 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:56:54.055320+0000 mgr.smithi047.fuhzzr (mgr.14162) 260 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:56:55.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:55 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:56:54.952569+0000 mon.smithi047 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.194:0/407082242' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:56:56.345 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:56:56.345 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:56:57.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:57 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:56:56.055654+0000 mgr.smithi047.fuhzzr (mgr.14162) 261 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:56:57.517 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:57.515+0000 7f579cc13700 1 -- 172.21.15.194:0/3555776305 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57980f6510 msgr2=0x7f57980f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:57.517 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:57.515+0000 7f579cc13700 1 --2- 172.21.15.194:0/3555776305 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57980f6510 0x7f57980f6930 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f5780009fb0 tx=0x7f5780009b30).stop 2024-01-18T06:56:57.517 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:57.515+0000 7f579cc13700 1 -- 172.21.15.194:0/3555776305 shutdown_connections 2024-01-18T06:56:57.517 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:57.515+0000 7f579cc13700 1 --2- 172.21.15.194:0/3555776305 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57980f6510 0x7f57980f6930 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:57.517 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:57.515+0000 7f579cc13700 1 -- 172.21.15.194:0/3555776305 >> 172.21.15.194:0/3555776305 conn(0x7f57980f1c70 msgr2=0x7f57980f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:56:57.517 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:57.516+0000 7f579cc13700 1 -- 172.21.15.194:0/3555776305 shutdown_connections 2024-01-18T06:56:57.517 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:57.516+0000 7f579cc13700 1 -- 172.21.15.194:0/3555776305 wait complete. 2024-01-18T06:56:57.517 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:57.516+0000 7f579cc13700 1 Processor -- start 2024-01-18T06:56:57.517 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:57.516+0000 7f579cc13700 1 -- start start 2024-01-18T06:56:57.517 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:57.516+0000 7f579cc13700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57980f6510 0x7f5798106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:56:57.517 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:57.516+0000 7f579cc13700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5798106c20 con 0x7f57980f6510 2024-01-18T06:56:57.518 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:57.516+0000 7f579659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57980f6510 0x7f5798106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:56:57.518 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:57.517+0000 7f579659c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57980f6510 0x7f5798106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:39048/0 (socket says 172.21.15.194:39048) 2024-01-18T06:56:57.518 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:57.517+0000 7f579659c700 1 -- 172.21.15.194:0/2847936489 learned_addr learned my addr 172.21.15.194:0/2847936489 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:56:57.518 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:57.517+0000 7f579659c700 1 -- 172.21.15.194:0/2847936489 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5780005040 con 0x7f57980f6510 2024-01-18T06:56:57.518 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:57.517+0000 7f579659c700 1 --2- 172.21.15.194:0/2847936489 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57980f6510 0x7f5798106730 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f5780014000 tx=0x7f578000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:56:57.518 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:57.517+0000 7f578effd700 1 -- 172.21.15.194:0/2847936489 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57800095f0 con 0x7f57980f6510 2024-01-18T06:56:57.518 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:57.517+0000 7f579cc13700 1 -- 172.21.15.194:0/2847936489 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5798106e20 con 0x7f57980f6510 2024-01-18T06:56:57.518 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:57.517+0000 7f578effd700 1 -- 172.21.15.194:0/2847936489 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5780007e70 con 0x7f57980f6510 2024-01-18T06:56:57.518 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:57.517+0000 7f578effd700 1 -- 172.21.15.194:0/2847936489 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5780014ea0 con 0x7f57980f6510 2024-01-18T06:56:57.518 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:57.517+0000 7f579cc13700 1 -- 172.21.15.194:0/2847936489 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57981072b0 con 0x7f57980f6510 2024-01-18T06:56:57.519 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:57.518+0000 7f579cc13700 1 -- 172.21.15.194:0/2847936489 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5798045ad0 con 0x7f57980f6510 2024-01-18T06:56:57.519 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:57.518+0000 7f578effd700 1 -- 172.21.15.194:0/2847936489 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5780025070 con 0x7f57980f6510 2024-01-18T06:56:57.519 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:57.518+0000 7f578effd700 1 --2- 172.21.15.194:0/2847936489 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f578402ff60 0x7f5784032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:56:57.519 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:57.518+0000 7f578effd700 1 -- 172.21.15.194:0/2847936489 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5780040930 con 0x7f57980f6510 2024-01-18T06:56:57.519 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:57.518+0000 7f5795d9b700 1 --2- 172.21.15.194:0/2847936489 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f578402ff60 0x7f5784032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:56:57.519 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:57.519+0000 7f5795d9b700 1 --2- 172.21.15.194:0/2847936489 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f578402ff60 0x7f5784032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f57880096a0 tx=0x7f5788006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:56:57.522 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:57.523+0000 7f578effd700 1 -- 172.21.15.194:0/2847936489 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f578000f4e0 con 0x7f57980f6510 2024-01-18T06:56:57.729 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:57.730+0000 7f579cc13700 1 -- 172.21.15.194:0/2847936489 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f57980fbf60 con 0x7f57980f6510 2024-01-18T06:56:57.730 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:57.730+0000 7f578effd700 1 -- 172.21.15.194:0/2847936489 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f578000f300 con 0x7f57980f6510 2024-01-18T06:56:57.730 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:56:57.730 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:56:57.732 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:57.732+0000 7f579cc13700 1 -- 172.21.15.194:0/2847936489 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f578402ff60 msgr2=0x7f5784032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:57.732 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:57.732+0000 7f579cc13700 1 --2- 172.21.15.194:0/2847936489 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f578402ff60 0x7f5784032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f57880096a0 tx=0x7f5788006b40).stop 2024-01-18T06:56:57.732 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:57.732+0000 7f579cc13700 1 -- 172.21.15.194:0/2847936489 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57980f6510 msgr2=0x7f5798106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:56:57.732 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:57.732+0000 7f579cc13700 1 --2- 172.21.15.194:0/2847936489 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57980f6510 0x7f5798106730 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f5780014000 tx=0x7f578000b040).stop 2024-01-18T06:56:57.732 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:57.732+0000 7f579cc13700 1 -- 172.21.15.194:0/2847936489 shutdown_connections 2024-01-18T06:56:57.732 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:57.732+0000 7f579cc13700 1 --2- 172.21.15.194:0/2847936489 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f578402ff60 0x7f5784032420 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:57.732 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:57.732+0000 7f579cc13700 1 --2- 172.21.15.194:0/2847936489 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f57980f6510 0x7f5798106730 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:56:57.732 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:57.732+0000 7f579cc13700 1 -- 172.21.15.194:0/2847936489 >> 172.21.15.194:0/2847936489 conn(0x7f57980f1c70 msgr2=0x7f57980f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:56:57.732 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:57.732+0000 7f579cc13700 1 -- 172.21.15.194:0/2847936489 shutdown_connections 2024-01-18T06:56:57.732 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:56:57.732+0000 7f579cc13700 1 -- 172.21.15.194:0/2847936489 wait complete. 2024-01-18T06:56:57.732 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:56:58.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:58 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:56:57.731238+0000 mon.smithi047 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.194:0/2847936489' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:56:59.126 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:56:59.127 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:56:59.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:56:59 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:56:58.055984+0000 mgr.smithi047.fuhzzr (mgr.14162) 262 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:57:00.365 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:00.365+0000 7fb09ffff700 1 -- 172.21.15.194:0/771231967 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0a00f51a0 msgr2=0x7fb0a00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:57:00.365 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:00.365+0000 7fb09ffff700 1 --2- 172.21.15.194:0/771231967 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0a00f51a0 0x7fb0a00f55c0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7fb090009fb0 tx=0x7fb090009b30).stop 2024-01-18T06:57:00.366 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:00.365+0000 7fb09ffff700 1 -- 172.21.15.194:0/771231967 shutdown_connections 2024-01-18T06:57:00.366 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:00.365+0000 7fb09ffff700 1 --2- 172.21.15.194:0/771231967 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0a00f51a0 0x7fb0a00f55c0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:57:00.366 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:00.365+0000 7fb09ffff700 1 -- 172.21.15.194:0/771231967 >> 172.21.15.194:0/771231967 conn(0x7fb0a00f09a0 msgr2=0x7fb0a00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:57:00.366 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:00.365+0000 7fb09ffff700 1 -- 172.21.15.194:0/771231967 shutdown_connections 2024-01-18T06:57:00.366 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:00.365+0000 7fb09ffff700 1 -- 172.21.15.194:0/771231967 wait complete. 2024-01-18T06:57:00.367 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:00.366+0000 7fb09ffff700 1 Processor -- start 2024-01-18T06:57:00.367 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:00.366+0000 7fb09ffff700 1 -- start start 2024-01-18T06:57:00.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:00.366+0000 7fb09ffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0a00f51a0 0x7fb0a01091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:57:00.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:00.366+0000 7fb09ffff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a0060240 con 0x7fb0a00f51a0 2024-01-18T06:57:00.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:00.366+0000 7fb09effd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0a00f51a0 0x7fb0a01091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:57:00.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:00.366+0000 7fb09effd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0a00f51a0 0x7fb0a01091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:39072/0 (socket says 172.21.15.194:39072) 2024-01-18T06:57:00.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:00.366+0000 7fb09effd700 1 -- 172.21.15.194:0/239866056 learned_addr learned my addr 172.21.15.194:0/239866056 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:57:00.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:00.366+0000 7fb09effd700 1 -- 172.21.15.194:0/239866056 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb090005040 con 0x7fb0a00f51a0 2024-01-18T06:57:00.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:00.367+0000 7fb09effd700 1 --2- 172.21.15.194:0/239866056 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0a00f51a0 0x7fb0a01091b0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7fb090013510 tx=0x7fb09000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:57:00.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:00.367+0000 7fb0877fe700 1 -- 172.21.15.194:0/239866056 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb090009320 con 0x7fb0a00f51a0 2024-01-18T06:57:00.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:00.367+0000 7fb09ffff700 1 -- 172.21.15.194:0/239866056 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0a0060440 con 0x7fb0a00f51a0 2024-01-18T06:57:00.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:00.367+0000 7fb0877fe700 1 -- 172.21.15.194:0/239866056 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb090013f00 con 0x7fb0a00f51a0 2024-01-18T06:57:00.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:00.367+0000 7fb0877fe700 1 -- 172.21.15.194:0/239866056 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb090014dc0 con 0x7fb0a00f51a0 2024-01-18T06:57:00.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:00.367+0000 7fb09ffff700 1 -- 172.21.15.194:0/239866056 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0a0109860 con 0x7fb0a00f51a0 2024-01-18T06:57:00.369 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:00.368+0000 7fb0877fe700 1 -- 172.21.15.194:0/239866056 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb0900095f0 con 0x7fb0a00f51a0 2024-01-18T06:57:00.369 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:00.368+0000 7fb09ffff700 1 -- 172.21.15.194:0/239866056 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0a0045ad0 con 0x7fb0a00f51a0 2024-01-18T06:57:00.369 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:00.368+0000 7fb0877fe700 1 --2- 172.21.15.194:0/239866056 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb08802ffb0 0x7fb088032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:57:00.370 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:00.368+0000 7fb0877fe700 1 -- 172.21.15.194:0/239866056 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb090041070 con 0x7fb0a00f51a0 2024-01-18T06:57:00.370 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:00.368+0000 7fb09e7fc700 1 --2- 172.21.15.194:0/239866056 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb08802ffb0 0x7fb088032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:57:00.370 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:00.368+0000 7fb09e7fc700 1 --2- 172.21.15.194:0/239866056 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb08802ffb0 0x7fb088032470 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fb0940096a0 tx=0x7fb094006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:57:00.373 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:00.372+0000 7fb0877fe700 1 -- 172.21.15.194:0/239866056 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb09006d2b0 con 0x7fb0a00f51a0 2024-01-18T06:57:00.577 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:00.576+0000 7fb09ffff700 1 -- 172.21.15.194:0/239866056 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb0a010a080 con 0x7fb0a00f51a0 2024-01-18T06:57:00.577 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:00.577+0000 7fb0877fe700 1 -- 172.21.15.194:0/239866056 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb090014420 con 0x7fb0a00f51a0 2024-01-18T06:57:00.577 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:57:00.578 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:57:00.579 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:00.578+0000 7fb09ffff700 1 -- 172.21.15.194:0/239866056 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb08802ffb0 msgr2=0x7fb088032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:57:00.579 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:00.579+0000 7fb09ffff700 1 --2- 172.21.15.194:0/239866056 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb08802ffb0 0x7fb088032470 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fb0940096a0 tx=0x7fb094006b40).stop 2024-01-18T06:57:00.579 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:00.579+0000 7fb09ffff700 1 -- 172.21.15.194:0/239866056 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0a00f51a0 msgr2=0x7fb0a01091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:57:00.579 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:00.579+0000 7fb09ffff700 1 --2- 172.21.15.194:0/239866056 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0a00f51a0 0x7fb0a01091b0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7fb090013510 tx=0x7fb09000b040).stop 2024-01-18T06:57:00.579 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:00.579+0000 7fb09ffff700 1 -- 172.21.15.194:0/239866056 shutdown_connections 2024-01-18T06:57:00.579 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:00.579+0000 7fb09ffff700 1 --2- 172.21.15.194:0/239866056 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fb08802ffb0 0x7fb088032470 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:57:00.579 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:00.579+0000 7fb09ffff700 1 --2- 172.21.15.194:0/239866056 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0a00f51a0 0x7fb0a01091b0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:57:00.579 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:00.579+0000 7fb09ffff700 1 -- 172.21.15.194:0/239866056 >> 172.21.15.194:0/239866056 conn(0x7fb0a00f09a0 msgr2=0x7fb0a00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:57:00.580 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:00.579+0000 7fb09ffff700 1 -- 172.21.15.194:0/239866056 shutdown_connections 2024-01-18T06:57:00.580 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:00.579+0000 7fb09ffff700 1 -- 172.21.15.194:0/239866056 wait complete. 2024-01-18T06:57:00.580 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:57:01.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:57:01 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:57:00.056329+0000 mgr.smithi047.fuhzzr (mgr.14162) 263 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:57:01.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:57:01 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:57:00.579031+0000 mon.smithi047 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.194:0/239866056' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:57:01.962 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:57:01.962 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:57:03.116 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:03.114+0000 7fc7ed0e7700 1 -- 172.21.15.194:0/3369113825 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7e80f6510 msgr2=0x7fc7e80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:57:03.116 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:03.114+0000 7fc7ed0e7700 1 --2- 172.21.15.194:0/3369113825 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7e80f6510 0x7fc7e80f6930 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7fc7d0009fb0 tx=0x7fc7d0009b30).stop 2024-01-18T06:57:03.116 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:03.114+0000 7fc7ed0e7700 1 -- 172.21.15.194:0/3369113825 shutdown_connections 2024-01-18T06:57:03.116 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:03.114+0000 7fc7ed0e7700 1 --2- 172.21.15.194:0/3369113825 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7e80f6510 0x7fc7e80f6930 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:57:03.116 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:03.114+0000 7fc7ed0e7700 1 -- 172.21.15.194:0/3369113825 >> 172.21.15.194:0/3369113825 conn(0x7fc7e80f1c70 msgr2=0x7fc7e80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:57:03.116 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:03.114+0000 7fc7ed0e7700 1 -- 172.21.15.194:0/3369113825 shutdown_connections 2024-01-18T06:57:03.116 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:03.114+0000 7fc7ed0e7700 1 -- 172.21.15.194:0/3369113825 wait complete. 2024-01-18T06:57:03.117 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:03.115+0000 7fc7ed0e7700 1 Processor -- start 2024-01-18T06:57:03.117 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:03.115+0000 7fc7ed0e7700 1 -- start start 2024-01-18T06:57:03.117 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:03.115+0000 7fc7ed0e7700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7e80f6510 0x7fc7e8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:57:03.117 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:03.115+0000 7fc7ed0e7700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e8106c20 con 0x7fc7e80f6510 2024-01-18T06:57:03.117 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:03.115+0000 7fc7e6d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7e80f6510 0x7fc7e8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:57:03.117 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:03.115+0000 7fc7e6d9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7e80f6510 0x7fc7e8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:39090/0 (socket says 172.21.15.194:39090) 2024-01-18T06:57:03.117 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:03.115+0000 7fc7e6d9d700 1 -- 172.21.15.194:0/4116266509 learned_addr learned my addr 172.21.15.194:0/4116266509 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:57:03.117 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:03.115+0000 7fc7e6d9d700 1 -- 172.21.15.194:0/4116266509 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d0005040 con 0x7fc7e80f6510 2024-01-18T06:57:03.117 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:03.115+0000 7fc7e6d9d700 1 --2- 172.21.15.194:0/4116266509 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7e80f6510 0x7fc7e8106730 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7fc7d0014000 tx=0x7fc7d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:57:03.117 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:03.116+0000 7fc7df7fe700 1 -- 172.21.15.194:0/4116266509 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7d00095f0 con 0x7fc7e80f6510 2024-01-18T06:57:03.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:03.116+0000 7fc7ed0e7700 1 -- 172.21.15.194:0/4116266509 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7e8106e20 con 0x7fc7e80f6510 2024-01-18T06:57:03.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:03.116+0000 7fc7df7fe700 1 -- 172.21.15.194:0/4116266509 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc7d0007e70 con 0x7fc7e80f6510 2024-01-18T06:57:03.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:03.116+0000 7fc7df7fe700 1 -- 172.21.15.194:0/4116266509 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7d0014ea0 con 0x7fc7e80f6510 2024-01-18T06:57:03.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:03.116+0000 7fc7ed0e7700 1 -- 172.21.15.194:0/4116266509 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7e81072b0 con 0x7fc7e80f6510 2024-01-18T06:57:03.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:03.116+0000 7fc7df7fe700 1 -- 172.21.15.194:0/4116266509 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc7d00079c0 con 0x7fc7e80f6510 2024-01-18T06:57:03.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:03.116+0000 7fc7ed0e7700 1 -- 172.21.15.194:0/4116266509 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7e8045ad0 con 0x7fc7e80f6510 2024-01-18T06:57:03.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:03.117+0000 7fc7df7fe700 1 --2- 172.21.15.194:0/4116266509 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc7d402ffb0 0x7fc7d4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:57:03.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:03.117+0000 7fc7df7fe700 1 -- 172.21.15.194:0/4116266509 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc7d00408b0 con 0x7fc7e80f6510 2024-01-18T06:57:03.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:03.117+0000 7fc7e659c700 1 --2- 172.21.15.194:0/4116266509 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc7d402ffb0 0x7fc7d4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:57:03.120 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:03.117+0000 7fc7e659c700 1 --2- 172.21.15.194:0/4116266509 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc7d402ffb0 0x7fc7d4032470 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fc7d80096a0 tx=0x7fc7d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:57:03.123 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:03.121+0000 7fc7df7fe700 1 -- 172.21.15.194:0/4116266509 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc7d000f950 con 0x7fc7e80f6510 2024-01-18T06:57:03.328 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:03.327+0000 7fc7ed0e7700 1 -- 172.21.15.194:0/4116266509 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc7e80fbf60 con 0x7fc7e80f6510 2024-01-18T06:57:03.328 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:03.328+0000 7fc7df7fe700 1 -- 172.21.15.194:0/4116266509 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc7d0010230 con 0x7fc7e80f6510 2024-01-18T06:57:03.328 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:57:03.328 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:57:03.330 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:03.329+0000 7fc7ed0e7700 1 -- 172.21.15.194:0/4116266509 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc7d402ffb0 msgr2=0x7fc7d4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:57:03.330 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:03.329+0000 7fc7ed0e7700 1 --2- 172.21.15.194:0/4116266509 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc7d402ffb0 0x7fc7d4032470 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fc7d80096a0 tx=0x7fc7d8006b40).stop 2024-01-18T06:57:03.330 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:03.329+0000 7fc7ed0e7700 1 -- 172.21.15.194:0/4116266509 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7e80f6510 msgr2=0x7fc7e8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:57:03.330 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:03.329+0000 7fc7ed0e7700 1 --2- 172.21.15.194:0/4116266509 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7e80f6510 0x7fc7e8106730 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7fc7d0014000 tx=0x7fc7d000b040).stop 2024-01-18T06:57:03.330 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:03.329+0000 7fc7ed0e7700 1 -- 172.21.15.194:0/4116266509 shutdown_connections 2024-01-18T06:57:03.330 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:03.329+0000 7fc7ed0e7700 1 --2- 172.21.15.194:0/4116266509 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fc7d402ffb0 0x7fc7d4032470 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:57:03.331 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:03.329+0000 7fc7ed0e7700 1 --2- 172.21.15.194:0/4116266509 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7e80f6510 0x7fc7e8106730 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:57:03.331 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:03.330+0000 7fc7ed0e7700 1 -- 172.21.15.194:0/4116266509 >> 172.21.15.194:0/4116266509 conn(0x7fc7e80f1c70 msgr2=0x7fc7e80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:57:03.331 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:03.330+0000 7fc7ed0e7700 1 -- 172.21.15.194:0/4116266509 shutdown_connections 2024-01-18T06:57:03.331 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:03.330+0000 7fc7ed0e7700 1 -- 172.21.15.194:0/4116266509 wait complete. 2024-01-18T06:57:03.331 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:57:03.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:57:03 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:57:02.056668+0000 mgr.smithi047.fuhzzr (mgr.14162) 264 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:57:04.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:57:04 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:57:03.329889+0000 mon.smithi047 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.194:0/4116266509' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:57:04.667 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:57:04.668 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:57:05.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:57:05 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:57:04.057028+0000 mgr.smithi047.fuhzzr (mgr.14162) 265 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:57:05.837 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:05.835+0000 7f0ce48ac700 1 -- 172.21.15.194:0/3128962253 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ce00f62a0 msgr2=0x7f0ce00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:57:05.837 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:05.836+0000 7f0ce48ac700 1 --2- 172.21.15.194:0/3128962253 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ce00f62a0 0x7f0ce00f66c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f0cd0009fb0 tx=0x7f0cd0009b30).stop 2024-01-18T06:57:05.838 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:05.836+0000 7f0ce48ac700 1 -- 172.21.15.194:0/3128962253 shutdown_connections 2024-01-18T06:57:05.838 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:05.836+0000 7f0ce48ac700 1 --2- 172.21.15.194:0/3128962253 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ce00f62a0 0x7f0ce00f66c0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:57:05.838 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:05.836+0000 7f0ce48ac700 1 -- 172.21.15.194:0/3128962253 >> 172.21.15.194:0/3128962253 conn(0x7f0ce00f1aa0 msgr2=0x7f0ce00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:57:05.838 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:05.836+0000 7f0ce48ac700 1 -- 172.21.15.194:0/3128962253 shutdown_connections 2024-01-18T06:57:05.838 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:05.836+0000 7f0ce48ac700 1 -- 172.21.15.194:0/3128962253 wait complete. 2024-01-18T06:57:05.838 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:05.836+0000 7f0ce48ac700 1 Processor -- start 2024-01-18T06:57:05.838 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:05.836+0000 7f0ce48ac700 1 -- start start 2024-01-18T06:57:05.838 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:05.836+0000 7f0ce48ac700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ce00f62a0 0x7f0ce01064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:57:05.838 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:05.836+0000 7f0ce48ac700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce01069a0 con 0x7f0ce00f62a0 2024-01-18T06:57:05.838 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:05.837+0000 7f0cdf7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ce00f62a0 0x7f0ce01064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:57:05.838 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:05.837+0000 7f0cdf7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ce00f62a0 0x7f0ce01064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:39116/0 (socket says 172.21.15.194:39116) 2024-01-18T06:57:05.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:05.837+0000 7f0cdf7fe700 1 -- 172.21.15.194:0/1368649931 learned_addr learned my addr 172.21.15.194:0/1368649931 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:57:05.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:05.837+0000 7f0cdf7fe700 1 -- 172.21.15.194:0/1368649931 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd0005040 con 0x7f0ce00f62a0 2024-01-18T06:57:05.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:05.837+0000 7f0cdf7fe700 1 --2- 172.21.15.194:0/1368649931 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ce00f62a0 0x7f0ce01064b0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f0cd0013510 tx=0x7f0cd000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:57:05.840 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:05.837+0000 7f0cc7fff700 1 -- 172.21.15.194:0/1368649931 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cd0009320 con 0x7f0ce00f62a0 2024-01-18T06:57:05.840 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:05.837+0000 7f0ce48ac700 1 -- 172.21.15.194:0/1368649931 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ce0106ba0 con 0x7f0ce00f62a0 2024-01-18T06:57:05.840 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:05.838+0000 7f0cc7fff700 1 -- 172.21.15.194:0/1368649931 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0cd0013f00 con 0x7f0ce00f62a0 2024-01-18T06:57:05.840 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:05.838+0000 7f0cc7fff700 1 -- 172.21.15.194:0/1368649931 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cd0014dc0 con 0x7f0ce00f62a0 2024-01-18T06:57:05.840 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:05.838+0000 7f0ce48ac700 1 -- 172.21.15.194:0/1368649931 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ce0107030 con 0x7f0ce00f62a0 2024-01-18T06:57:05.840 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:05.838+0000 7f0cc7fff700 1 -- 172.21.15.194:0/1368649931 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0cd0014420 con 0x7f0ce00f62a0 2024-01-18T06:57:05.840 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:05.838+0000 7f0ce48ac700 1 -- 172.21.15.194:0/1368649931 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ce0045ad0 con 0x7f0ce00f62a0 2024-01-18T06:57:05.841 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:05.838+0000 7f0cc7fff700 1 --2- 172.21.15.194:0/1368649931 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f0cc802ffb0 0x7f0cc8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:57:05.841 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:05.838+0000 7f0cc7fff700 1 -- 172.21.15.194:0/1368649931 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0cd0041110 con 0x7f0ce00f62a0 2024-01-18T06:57:05.841 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:05.839+0000 7f0cdeffd700 1 --2- 172.21.15.194:0/1368649931 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f0cc802ffb0 0x7f0cc8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:57:05.841 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:05.839+0000 7f0cdeffd700 1 --2- 172.21.15.194:0/1368649931 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f0cc802ffb0 0x7f0cc8032470 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f0cd40096a0 tx=0x7f0cd4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:57:05.843 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:05.844+0000 7f0cc7fff700 1 -- 172.21.15.194:0/1368649931 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0cd00095f0 con 0x7f0ce00f62a0 2024-01-18T06:57:06.048 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:06.048+0000 7f0ce48ac700 1 -- 172.21.15.194:0/1368649931 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0ce0107ee0 con 0x7f0ce00f62a0 2024-01-18T06:57:06.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:06.050+0000 7f0cc7fff700 1 -- 172.21.15.194:0/1368649931 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0cd001cb10 con 0x7f0ce00f62a0 2024-01-18T06:57:06.049 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:57:06.050 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:57:06.051 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:06.050+0000 7f0ce48ac700 1 -- 172.21.15.194:0/1368649931 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f0cc802ffb0 msgr2=0x7f0cc8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:57:06.051 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:06.050+0000 7f0ce48ac700 1 --2- 172.21.15.194:0/1368649931 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f0cc802ffb0 0x7f0cc8032470 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f0cd40096a0 tx=0x7f0cd4006b40).stop 2024-01-18T06:57:06.051 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:06.050+0000 7f0ce48ac700 1 -- 172.21.15.194:0/1368649931 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ce00f62a0 msgr2=0x7f0ce01064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:57:06.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:06.050+0000 7f0ce48ac700 1 --2- 172.21.15.194:0/1368649931 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ce00f62a0 0x7f0ce01064b0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f0cd0013510 tx=0x7f0cd000b040).stop 2024-01-18T06:57:06.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:06.050+0000 7f0ce48ac700 1 -- 172.21.15.194:0/1368649931 shutdown_connections 2024-01-18T06:57:06.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:06.050+0000 7f0ce48ac700 1 --2- 172.21.15.194:0/1368649931 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f0cc802ffb0 0x7f0cc8032470 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:57:06.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:06.050+0000 7f0ce48ac700 1 --2- 172.21.15.194:0/1368649931 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ce00f62a0 0x7f0ce01064b0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:57:06.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:06.050+0000 7f0ce48ac700 1 -- 172.21.15.194:0/1368649931 >> 172.21.15.194:0/1368649931 conn(0x7f0ce00f1aa0 msgr2=0x7f0ce00f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:57:06.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:06.050+0000 7f0ce48ac700 1 -- 172.21.15.194:0/1368649931 shutdown_connections 2024-01-18T06:57:06.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:06.050+0000 7f0ce48ac700 1 -- 172.21.15.194:0/1368649931 wait complete. 2024-01-18T06:57:06.052 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:57:06.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:57:06 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:57:06.050695+0000 mon.smithi047 (mon.0) 315 : audit [DBG] from='client.? 172.21.15.194:0/1368649931' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:57:07.460 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:57:07.460 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:57:07.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:57:07 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:57:06.057374+0000 mgr.smithi047.fuhzzr (mgr.14162) 266 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:57:08.647 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:08.645+0000 7fd2048b2700 1 -- 172.21.15.194:0/3161089273 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2000f5410 msgr2=0x7fd2000f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:57:08.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:08.645+0000 7fd2048b2700 1 --2- 172.21.15.194:0/3161089273 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2000f5410 0x7fd2000f5830 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7fd1f4009fb0 tx=0x7fd1f4009b30).stop 2024-01-18T06:57:08.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:08.645+0000 7fd2048b2700 1 -- 172.21.15.194:0/3161089273 shutdown_connections 2024-01-18T06:57:08.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:08.645+0000 7fd2048b2700 1 --2- 172.21.15.194:0/3161089273 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2000f5410 0x7fd2000f5830 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:57:08.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:08.645+0000 7fd2048b2700 1 -- 172.21.15.194:0/3161089273 >> 172.21.15.194:0/3161089273 conn(0x7fd2000f0b70 msgr2=0x7fd2000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:57:08.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:08.645+0000 7fd2048b2700 1 -- 172.21.15.194:0/3161089273 shutdown_connections 2024-01-18T06:57:08.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:08.645+0000 7fd2048b2700 1 -- 172.21.15.194:0/3161089273 wait complete. 2024-01-18T06:57:08.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:08.646+0000 7fd2048b2700 1 Processor -- start 2024-01-18T06:57:08.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:08.646+0000 7fd2048b2700 1 -- start start 2024-01-18T06:57:08.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:08.646+0000 7fd2048b2700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2000f5410 0x7fd20010b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:57:08.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:08.646+0000 7fd2048b2700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd200107240 con 0x7fd2000f5410 2024-01-18T06:57:08.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:08.646+0000 7fd1fe59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2000f5410 0x7fd20010b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:57:08.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:08.646+0000 7fd1fe59c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2000f5410 0x7fd20010b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:42898/0 (socket says 172.21.15.194:42898) 2024-01-18T06:57:08.650 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:08.646+0000 7fd1fe59c700 1 -- 172.21.15.194:0/1267030522 learned_addr learned my addr 172.21.15.194:0/1267030522 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:57:08.650 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:08.646+0000 7fd1fe59c700 1 -- 172.21.15.194:0/1267030522 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1f4005040 con 0x7fd2000f5410 2024-01-18T06:57:08.650 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:08.647+0000 7fd1fe59c700 1 --2- 172.21.15.194:0/1267030522 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2000f5410 0x7fd20010b850 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fd1f4013510 tx=0x7fd1f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:57:08.650 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:08.647+0000 7fd1eeffd700 1 -- 172.21.15.194:0/1267030522 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1f4009320 con 0x7fd2000f5410 2024-01-18T06:57:08.650 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:08.647+0000 7fd2048b2700 1 -- 172.21.15.194:0/1267030522 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd200060220 con 0x7fd2000f5410 2024-01-18T06:57:08.650 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:08.647+0000 7fd1eeffd700 1 -- 172.21.15.194:0/1267030522 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd1f4013f00 con 0x7fd2000f5410 2024-01-18T06:57:08.650 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:08.647+0000 7fd1eeffd700 1 -- 172.21.15.194:0/1267030522 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1f4014d90 con 0x7fd2000f5410 2024-01-18T06:57:08.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:08.647+0000 7fd2048b2700 1 -- 172.21.15.194:0/1267030522 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd20010bd40 con 0x7fd2000f5410 2024-01-18T06:57:08.652 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:08.648+0000 7fd1eeffd700 1 -- 172.21.15.194:0/1267030522 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd1f40263a0 con 0x7fd2000f5410 2024-01-18T06:57:08.652 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:08.648+0000 7fd1eeffd700 1 --2- 172.21.15.194:0/1267030522 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fd1e802ff60 0x7fd1e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:57:08.652 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:08.648+0000 7fd1eeffd700 1 -- 172.21.15.194:0/1267030522 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd1f40412f0 con 0x7fd2000f5410 2024-01-18T06:57:08.653 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:08.648+0000 7fd2048b2700 1 -- 172.21.15.194:0/1267030522 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd200045ad0 con 0x7fd2000f5410 2024-01-18T06:57:08.653 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:08.648+0000 7fd1fdd9b700 1 --2- 172.21.15.194:0/1267030522 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fd1e802ff60 0x7fd1e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:57:08.653 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:08.648+0000 7fd1fdd9b700 1 --2- 172.21.15.194:0/1267030522 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fd1e802ff60 0x7fd1e8032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fd1f00096a0 tx=0x7fd1f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:57:08.654 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:08.653+0000 7fd1eeffd700 1 -- 172.21.15.194:0/1267030522 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd1f4014420 con 0x7fd2000f5410 2024-01-18T06:57:08.858 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:08.858+0000 7fd2048b2700 1 -- 172.21.15.194:0/1267030522 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd20010c3e0 con 0x7fd2000f5410 2024-01-18T06:57:08.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:08.860+0000 7fd1eeffd700 1 -- 172.21.15.194:0/1267030522 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd1f402c090 con 0x7fd2000f5410 2024-01-18T06:57:08.859 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:57:08.859 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:57:08.861 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:08.860+0000 7fd2048b2700 1 -- 172.21.15.194:0/1267030522 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fd1e802ff60 msgr2=0x7fd1e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:57:08.861 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:08.860+0000 7fd2048b2700 1 --2- 172.21.15.194:0/1267030522 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fd1e802ff60 0x7fd1e8032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fd1f00096a0 tx=0x7fd1f0006b40).stop 2024-01-18T06:57:08.861 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:08.860+0000 7fd2048b2700 1 -- 172.21.15.194:0/1267030522 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2000f5410 msgr2=0x7fd20010b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:57:08.861 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:08.860+0000 7fd2048b2700 1 --2- 172.21.15.194:0/1267030522 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2000f5410 0x7fd20010b850 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fd1f4013510 tx=0x7fd1f400b040).stop 2024-01-18T06:57:08.861 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:08.860+0000 7fd2048b2700 1 -- 172.21.15.194:0/1267030522 shutdown_connections 2024-01-18T06:57:08.861 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:08.860+0000 7fd2048b2700 1 --2- 172.21.15.194:0/1267030522 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fd1e802ff60 0x7fd1e8032420 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:57:08.861 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:08.860+0000 7fd2048b2700 1 --2- 172.21.15.194:0/1267030522 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2000f5410 0x7fd20010b850 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:57:08.861 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:08.860+0000 7fd2048b2700 1 -- 172.21.15.194:0/1267030522 >> 172.21.15.194:0/1267030522 conn(0x7fd2000f0b70 msgr2=0x7fd2000f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:57:08.861 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:08.860+0000 7fd2048b2700 1 -- 172.21.15.194:0/1267030522 shutdown_connections 2024-01-18T06:57:08.862 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:08.861+0000 7fd2048b2700 1 -- 172.21.15.194:0/1267030522 wait complete. 2024-01-18T06:57:08.862 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:57:09.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:57:08 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:57:08.057699+0000 mgr.smithi047.fuhzzr (mgr.14162) 267 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:57:09.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:57:08 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:57:08.860602+0000 mon.smithi047 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.194:0/1267030522' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:57:10.268 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:57:10.269 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:57:11.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:11.413+0000 7f447616b700 1 -- 172.21.15.194:0/3435337411 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44700f62a0 msgr2=0x7f44700f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:57:11.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:11.413+0000 7f447616b700 1 --2- 172.21.15.194:0/3435337411 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44700f62a0 0x7f44700f66c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f4460009fb0 tx=0x7f4460009b30).stop 2024-01-18T06:57:11.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:11.413+0000 7f447616b700 1 -- 172.21.15.194:0/3435337411 shutdown_connections 2024-01-18T06:57:11.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:11.413+0000 7f447616b700 1 --2- 172.21.15.194:0/3435337411 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44700f62a0 0x7f44700f66c0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:57:11.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:11.413+0000 7f447616b700 1 -- 172.21.15.194:0/3435337411 >> 172.21.15.194:0/3435337411 conn(0x7f44700f1aa0 msgr2=0x7f44700f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:57:11.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:11.413+0000 7f447616b700 1 -- 172.21.15.194:0/3435337411 shutdown_connections 2024-01-18T06:57:11.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:11.413+0000 7f447616b700 1 -- 172.21.15.194:0/3435337411 wait complete. 2024-01-18T06:57:11.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:11.413+0000 7f447616b700 1 Processor -- start 2024-01-18T06:57:11.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:11.413+0000 7f447616b700 1 -- start start 2024-01-18T06:57:11.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:11.414+0000 7f447616b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44700f62a0 0x7f44701064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:57:11.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:11.414+0000 7f447616b700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44701069a0 con 0x7f44700f62a0 2024-01-18T06:57:11.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:11.414+0000 7f4475169700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44700f62a0 0x7f44701064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:57:11.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:11.414+0000 7f4475169700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44700f62a0 0x7f44701064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:42916/0 (socket says 172.21.15.194:42916) 2024-01-18T06:57:11.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:11.414+0000 7f4475169700 1 -- 172.21.15.194:0/2143336920 learned_addr learned my addr 172.21.15.194:0/2143336920 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:57:11.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:11.414+0000 7f4475169700 1 -- 172.21.15.194:0/2143336920 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4460005040 con 0x7f44700f62a0 2024-01-18T06:57:11.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:11.414+0000 7f4475169700 1 --2- 172.21.15.194:0/2143336920 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44700f62a0 0x7f44701064b0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f4460013510 tx=0x7f446000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:57:11.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:11.415+0000 7f4465ffb700 1 -- 172.21.15.194:0/2143336920 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4460009320 con 0x7f44700f62a0 2024-01-18T06:57:11.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:11.415+0000 7f447616b700 1 -- 172.21.15.194:0/2143336920 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4470106ba0 con 0x7f44700f62a0 2024-01-18T06:57:11.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:11.415+0000 7f4465ffb700 1 -- 172.21.15.194:0/2143336920 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4460013f00 con 0x7f44700f62a0 2024-01-18T06:57:11.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:11.415+0000 7f4465ffb700 1 -- 172.21.15.194:0/2143336920 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4460014dc0 con 0x7f44700f62a0 2024-01-18T06:57:11.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:11.415+0000 7f447616b700 1 -- 172.21.15.194:0/2143336920 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4470107030 con 0x7f44700f62a0 2024-01-18T06:57:11.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:11.416+0000 7f4465ffb700 1 -- 172.21.15.194:0/2143336920 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4460014420 con 0x7f44700f62a0 2024-01-18T06:57:11.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:11.416+0000 7f447616b700 1 -- 172.21.15.194:0/2143336920 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4470045ad0 con 0x7f44700f62a0 2024-01-18T06:57:11.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:11.416+0000 7f4465ffb700 1 --2- 172.21.15.194:0/2143336920 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f445c02ff60 0x7f445c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:57:11.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:11.416+0000 7f4465ffb700 1 -- 172.21.15.194:0/2143336920 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4460041030 con 0x7f44700f62a0 2024-01-18T06:57:11.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:11.416+0000 7f4474968700 1 --2- 172.21.15.194:0/2143336920 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f445c02ff60 0x7f445c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:57:11.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:11.416+0000 7f4474968700 1 --2- 172.21.15.194:0/2143336920 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f445c02ff60 0x7f445c032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f446c0096a0 tx=0x7f446c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:57:11.420 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:11.420+0000 7f4465ffb700 1 -- 172.21.15.194:0/2143336920 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f44600095f0 con 0x7f44700f62a0 2024-01-18T06:57:11.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:57:11 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:57:10.058038+0000 mgr.smithi047.fuhzzr (mgr.14162) 268 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:57:11.626 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:11.626+0000 7f447616b700 1 -- 172.21.15.194:0/2143336920 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4470107ee0 con 0x7f44700f62a0 2024-01-18T06:57:11.627 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:11.627+0000 7f4465ffb700 1 -- 172.21.15.194:0/2143336920 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f446001cb10 con 0x7f44700f62a0 2024-01-18T06:57:11.627 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:57:11.627 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:57:11.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:11.629+0000 7f447616b700 1 -- 172.21.15.194:0/2143336920 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f445c02ff60 msgr2=0x7f445c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:57:11.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:11.629+0000 7f447616b700 1 --2- 172.21.15.194:0/2143336920 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f445c02ff60 0x7f445c032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f446c0096a0 tx=0x7f446c006b40).stop 2024-01-18T06:57:11.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:11.629+0000 7f447616b700 1 -- 172.21.15.194:0/2143336920 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44700f62a0 msgr2=0x7f44701064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:57:11.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:11.629+0000 7f447616b700 1 --2- 172.21.15.194:0/2143336920 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44700f62a0 0x7f44701064b0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f4460013510 tx=0x7f446000b040).stop 2024-01-18T06:57:11.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:11.629+0000 7f447616b700 1 -- 172.21.15.194:0/2143336920 shutdown_connections 2024-01-18T06:57:11.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:11.629+0000 7f447616b700 1 --2- 172.21.15.194:0/2143336920 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f445c02ff60 0x7f445c032420 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:57:11.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:11.629+0000 7f447616b700 1 --2- 172.21.15.194:0/2143336920 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f44700f62a0 0x7f44701064b0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:57:11.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:11.629+0000 7f447616b700 1 -- 172.21.15.194:0/2143336920 >> 172.21.15.194:0/2143336920 conn(0x7f44700f1aa0 msgr2=0x7f44700f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:57:11.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:11.629+0000 7f447616b700 1 -- 172.21.15.194:0/2143336920 shutdown_connections 2024-01-18T06:57:11.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:11.629+0000 7f447616b700 1 -- 172.21.15.194:0/2143336920 wait complete. 2024-01-18T06:57:11.630 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:57:12.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:57:12 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:57:11.628321+0000 mon.smithi047 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.194:0/2143336920' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:57:13.015 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:57:13.016 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:57:13.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:57:13 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:57:12.058432+0000 mgr.smithi047.fuhzzr (mgr.14162) 269 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:57:14.141 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:14.141+0000 7f757e8b3700 1 -- 172.21.15.194:0/3262725415 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75780f5410 msgr2=0x7f75780f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:57:14.141 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:14.141+0000 7f757e8b3700 1 --2- 172.21.15.194:0/3262725415 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75780f5410 0x7f75780f5830 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f756c009fb0 tx=0x7f756c009b30).stop 2024-01-18T06:57:14.141 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:14.141+0000 7f757e8b3700 1 -- 172.21.15.194:0/3262725415 shutdown_connections 2024-01-18T06:57:14.141 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:14.141+0000 7f757e8b3700 1 --2- 172.21.15.194:0/3262725415 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75780f5410 0x7f75780f5830 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:57:14.142 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:14.141+0000 7f757e8b3700 1 -- 172.21.15.194:0/3262725415 >> 172.21.15.194:0/3262725415 conn(0x7f75780f0b70 msgr2=0x7f75780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:57:14.142 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:14.141+0000 7f757e8b3700 1 -- 172.21.15.194:0/3262725415 shutdown_connections 2024-01-18T06:57:14.142 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:14.141+0000 7f757e8b3700 1 -- 172.21.15.194:0/3262725415 wait complete. 2024-01-18T06:57:14.142 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:14.141+0000 7f757e8b3700 1 Processor -- start 2024-01-18T06:57:14.142 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:14.142+0000 7f757e8b3700 1 -- start start 2024-01-18T06:57:14.142 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:14.142+0000 7f757e8b3700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75780f5410 0x7f7578108910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:57:14.142 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:14.142+0000 7f757e8b3700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7578108e00 con 0x7f75780f5410 2024-01-18T06:57:14.142 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:14.142+0000 7f7577fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75780f5410 0x7f7578108910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:57:14.142 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:14.142+0000 7f7577fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75780f5410 0x7f7578108910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:42928/0 (socket says 172.21.15.194:42928) 2024-01-18T06:57:14.142 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:14.142+0000 7f7577fff700 1 -- 172.21.15.194:0/1983380981 learned_addr learned my addr 172.21.15.194:0/1983380981 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:57:14.144 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:14.142+0000 7f7577fff700 1 -- 172.21.15.194:0/1983380981 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f756c005040 con 0x7f75780f5410 2024-01-18T06:57:14.144 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:14.142+0000 7f7577fff700 1 --2- 172.21.15.194:0/1983380981 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75780f5410 0x7f7578108910 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f756c013510 tx=0x7f756c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:57:14.144 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:14.143+0000 7f7574ff9700 1 -- 172.21.15.194:0/1983380981 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f756c009320 con 0x7f75780f5410 2024-01-18T06:57:14.144 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:14.143+0000 7f7574ff9700 1 -- 172.21.15.194:0/1983380981 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f756c013f00 con 0x7f75780f5410 2024-01-18T06:57:14.144 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:14.143+0000 7f757e8b3700 1 -- 172.21.15.194:0/1983380981 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7578109000 con 0x7f75780f5410 2024-01-18T06:57:14.144 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:14.143+0000 7f7574ff9700 1 -- 172.21.15.194:0/1983380981 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f756c014d90 con 0x7f75780f5410 2024-01-18T06:57:14.144 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:14.143+0000 7f757e8b3700 1 -- 172.21.15.194:0/1983380981 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7578109490 con 0x7f75780f5410 2024-01-18T06:57:14.144 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:14.143+0000 7f7574ff9700 1 -- 172.21.15.194:0/1983380981 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f756c0095f0 con 0x7f75780f5410 2024-01-18T06:57:14.144 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:14.143+0000 7f757e8b3700 1 -- 172.21.15.194:0/1983380981 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7578045ad0 con 0x7f75780f5410 2024-01-18T06:57:14.144 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:14.144+0000 7f7574ff9700 1 --2- 172.21.15.194:0/1983380981 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f756002ffb0 0x7f7560032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:57:14.144 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:14.144+0000 7f7574ff9700 1 -- 172.21.15.194:0/1983380981 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f756c040ff0 con 0x7f75780f5410 2024-01-18T06:57:14.144 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:14.144+0000 7f75777fe700 1 --2- 172.21.15.194:0/1983380981 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f756002ffb0 0x7f7560032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:57:14.145 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:14.144+0000 7f75777fe700 1 --2- 172.21.15.194:0/1983380981 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f756002ffb0 0x7f7560032470 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f75680096a0 tx=0x7f7568006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:57:14.148 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:14.148+0000 7f7574ff9700 1 -- 172.21.15.194:0/1983380981 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f756c06d2b0 con 0x7f75780f5410 2024-01-18T06:57:14.354 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:14.355+0000 7f757e8b3700 1 -- 172.21.15.194:0/1983380981 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f75780fae60 con 0x7f75780f5410 2024-01-18T06:57:14.355 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:14.355+0000 7f7574ff9700 1 -- 172.21.15.194:0/1983380981 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f756c014420 con 0x7f75780f5410 2024-01-18T06:57:14.355 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:57:14.355 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:57:14.357 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:14.357+0000 7f757e8b3700 1 -- 172.21.15.194:0/1983380981 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f756002ffb0 msgr2=0x7f7560032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:57:14.357 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:14.357+0000 7f757e8b3700 1 --2- 172.21.15.194:0/1983380981 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f756002ffb0 0x7f7560032470 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f75680096a0 tx=0x7f7568006b40).stop 2024-01-18T06:57:14.357 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:14.357+0000 7f757e8b3700 1 -- 172.21.15.194:0/1983380981 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75780f5410 msgr2=0x7f7578108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:57:14.357 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:14.357+0000 7f757e8b3700 1 --2- 172.21.15.194:0/1983380981 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75780f5410 0x7f7578108910 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f756c013510 tx=0x7f756c00b040).stop 2024-01-18T06:57:14.357 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:14.357+0000 7f757e8b3700 1 -- 172.21.15.194:0/1983380981 shutdown_connections 2024-01-18T06:57:14.357 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:14.357+0000 7f757e8b3700 1 --2- 172.21.15.194:0/1983380981 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f756002ffb0 0x7f7560032470 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:57:14.357 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:14.357+0000 7f757e8b3700 1 --2- 172.21.15.194:0/1983380981 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75780f5410 0x7f7578108910 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:57:14.357 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:14.357+0000 7f757e8b3700 1 -- 172.21.15.194:0/1983380981 >> 172.21.15.194:0/1983380981 conn(0x7f75780f0b70 msgr2=0x7f757805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:57:14.357 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:14.357+0000 7f757e8b3700 1 -- 172.21.15.194:0/1983380981 shutdown_connections 2024-01-18T06:57:14.358 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:14.357+0000 7f757e8b3700 1 -- 172.21.15.194:0/1983380981 wait complete. 2024-01-18T06:57:14.358 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:57:15.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:57:15 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:57:14.058777+0000 mgr.smithi047.fuhzzr (mgr.14162) 270 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:57:15.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:57:15 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:57:14.356370+0000 mon.smithi047 (mon.0) 318 : audit [DBG] from='client.? 172.21.15.194:0/1983380981' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:57:15.735 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:57:15.735 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:57:16.891 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:16.888+0000 7fd53ef99700 1 -- 172.21.15.194:0/4051289256 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5380f62a0 msgr2=0x7fd5380f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:57:16.892 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:16.888+0000 7fd53ef99700 1 --2- 172.21.15.194:0/4051289256 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5380f62a0 0x7fd5380f66c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7fd534009fb0 tx=0x7fd534009b30).stop 2024-01-18T06:57:16.892 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:16.888+0000 7fd53ef99700 1 -- 172.21.15.194:0/4051289256 shutdown_connections 2024-01-18T06:57:16.892 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:16.888+0000 7fd53ef99700 1 --2- 172.21.15.194:0/4051289256 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5380f62a0 0x7fd5380f66c0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:57:16.892 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:16.888+0000 7fd53ef99700 1 -- 172.21.15.194:0/4051289256 >> 172.21.15.194:0/4051289256 conn(0x7fd5380f1aa0 msgr2=0x7fd5380f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:57:16.893 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:16.888+0000 7fd53ef99700 1 -- 172.21.15.194:0/4051289256 shutdown_connections 2024-01-18T06:57:16.893 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:16.888+0000 7fd53ef99700 1 -- 172.21.15.194:0/4051289256 wait complete. 2024-01-18T06:57:16.893 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:16.888+0000 7fd53ef99700 1 Processor -- start 2024-01-18T06:57:16.893 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:16.888+0000 7fd53ef99700 1 -- start start 2024-01-18T06:57:16.893 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:16.889+0000 7fd53ef99700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5380f62a0 0x7fd5381064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:57:16.893 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:16.889+0000 7fd53ef99700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5381069a0 con 0x7fd5380f62a0 2024-01-18T06:57:16.893 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:16.889+0000 7fd53df97700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5380f62a0 0x7fd5381064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:57:16.894 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:16.889+0000 7fd53df97700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5380f62a0 0x7fd5381064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:42934/0 (socket says 172.21.15.194:42934) 2024-01-18T06:57:16.894 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:16.889+0000 7fd53df97700 1 -- 172.21.15.194:0/3307798859 learned_addr learned my addr 172.21.15.194:0/3307798859 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:57:16.894 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:16.889+0000 7fd53df97700 1 -- 172.21.15.194:0/3307798859 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd534005040 con 0x7fd5380f62a0 2024-01-18T06:57:16.894 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:16.889+0000 7fd53df97700 1 --2- 172.21.15.194:0/3307798859 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5380f62a0 0x7fd5381064b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fd534013510 tx=0x7fd53400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:57:16.894 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:16.889+0000 7fd52e7fc700 1 -- 172.21.15.194:0/3307798859 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd534009320 con 0x7fd5380f62a0 2024-01-18T06:57:16.894 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:16.890+0000 7fd53ef99700 1 -- 172.21.15.194:0/3307798859 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd538106ba0 con 0x7fd5380f62a0 2024-01-18T06:57:16.895 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:16.890+0000 7fd52e7fc700 1 -- 172.21.15.194:0/3307798859 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd534013f00 con 0x7fd5380f62a0 2024-01-18T06:57:16.895 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:16.890+0000 7fd52e7fc700 1 -- 172.21.15.194:0/3307798859 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd534014dc0 con 0x7fd5380f62a0 2024-01-18T06:57:16.895 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:16.890+0000 7fd53ef99700 1 -- 172.21.15.194:0/3307798859 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd538107030 con 0x7fd5380f62a0 2024-01-18T06:57:16.895 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:16.890+0000 7fd52e7fc700 1 -- 172.21.15.194:0/3307798859 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd5340095f0 con 0x7fd5380f62a0 2024-01-18T06:57:16.895 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:16.890+0000 7fd53ef99700 1 -- 172.21.15.194:0/3307798859 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd538045ad0 con 0x7fd5380f62a0 2024-01-18T06:57:16.895 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:16.890+0000 7fd52e7fc700 1 --2- 172.21.15.194:0/3307798859 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fd52402ff60 0x7fd524032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:57:16.895 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:16.891+0000 7fd52e7fc700 1 -- 172.21.15.194:0/3307798859 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd5340410c0 con 0x7fd5380f62a0 2024-01-18T06:57:16.896 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:16.891+0000 7fd53d796700 1 --2- 172.21.15.194:0/3307798859 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fd52402ff60 0x7fd524032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:57:16.896 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:16.891+0000 7fd53d796700 1 --2- 172.21.15.194:0/3307798859 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fd52402ff60 0x7fd524032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fd5280096a0 tx=0x7fd528006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:57:16.896 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:16.896+0000 7fd52e7fc700 1 -- 172.21.15.194:0/3307798859 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd53401c790 con 0x7fd5380f62a0 2024-01-18T06:57:17.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:17.100+0000 7fd53ef99700 1 -- 172.21.15.194:0/3307798859 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd538107ee0 con 0x7fd5380f62a0 2024-01-18T06:57:17.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:17.101+0000 7fd52e7fc700 1 -- 172.21.15.194:0/3307798859 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd534010230 con 0x7fd5380f62a0 2024-01-18T06:57:17.100 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:57:17.101 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:57:17.102 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:17.103+0000 7fd53ef99700 1 -- 172.21.15.194:0/3307798859 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fd52402ff60 msgr2=0x7fd524032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:57:17.102 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:17.103+0000 7fd53ef99700 1 --2- 172.21.15.194:0/3307798859 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fd52402ff60 0x7fd524032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fd5280096a0 tx=0x7fd528006b40).stop 2024-01-18T06:57:17.102 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:17.103+0000 7fd53ef99700 1 -- 172.21.15.194:0/3307798859 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5380f62a0 msgr2=0x7fd5381064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:57:17.103 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:17.103+0000 7fd53ef99700 1 --2- 172.21.15.194:0/3307798859 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5380f62a0 0x7fd5381064b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fd534013510 tx=0x7fd53400b040).stop 2024-01-18T06:57:17.103 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:17.103+0000 7fd53ef99700 1 -- 172.21.15.194:0/3307798859 shutdown_connections 2024-01-18T06:57:17.103 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:17.103+0000 7fd53ef99700 1 --2- 172.21.15.194:0/3307798859 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fd52402ff60 0x7fd524032420 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:57:17.103 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:17.103+0000 7fd53ef99700 1 --2- 172.21.15.194:0/3307798859 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5380f62a0 0x7fd5381064b0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:57:17.103 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:17.103+0000 7fd53ef99700 1 -- 172.21.15.194:0/3307798859 >> 172.21.15.194:0/3307798859 conn(0x7fd5380f1aa0 msgr2=0x7fd5380f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:57:17.103 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:17.103+0000 7fd53ef99700 1 -- 172.21.15.194:0/3307798859 shutdown_connections 2024-01-18T06:57:17.103 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:17.103+0000 7fd53ef99700 1 -- 172.21.15.194:0/3307798859 wait complete. 2024-01-18T06:57:17.103 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:57:17.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:57:17 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:57:16.059153+0000 mgr.smithi047.fuhzzr (mgr.14162) 271 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:57:17.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:57:17 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:57:17.101835+0000 mon.smithi047 (mon.0) 319 : audit [DBG] from='client.? 172.21.15.194:0/3307798859' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:57:18.481 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:57:18.481 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:57:19.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:57:18 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:57:18.059550+0000 mgr.smithi047.fuhzzr (mgr.14162) 272 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:57:19.658 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:19.658+0000 7f544f958700 1 -- 172.21.15.194:0/2947430838 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54480f5410 msgr2=0x7f54480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:57:19.658 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:19.658+0000 7f544f958700 1 --2- 172.21.15.194:0/2947430838 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54480f5410 0x7f54480f5830 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f5438009fb0 tx=0x7f5438009b30).stop 2024-01-18T06:57:19.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:19.658+0000 7f544f958700 1 -- 172.21.15.194:0/2947430838 shutdown_connections 2024-01-18T06:57:19.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:19.658+0000 7f544f958700 1 --2- 172.21.15.194:0/2947430838 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54480f5410 0x7f54480f5830 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:57:19.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:19.658+0000 7f544f958700 1 -- 172.21.15.194:0/2947430838 >> 172.21.15.194:0/2947430838 conn(0x7f54480f0b70 msgr2=0x7f54480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:57:19.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:19.658+0000 7f544f958700 1 -- 172.21.15.194:0/2947430838 shutdown_connections 2024-01-18T06:57:19.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:19.658+0000 7f544f958700 1 -- 172.21.15.194:0/2947430838 wait complete. 2024-01-18T06:57:19.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:19.659+0000 7f544f958700 1 Processor -- start 2024-01-18T06:57:19.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:19.659+0000 7f544f958700 1 -- start start 2024-01-18T06:57:19.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:19.659+0000 7f544f958700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54480f5410 0x7f5448106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:57:19.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:19.659+0000 7f544f958700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5448106bf0 con 0x7f54480f5410 2024-01-18T06:57:19.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:19.659+0000 7f544d6f4700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54480f5410 0x7f5448106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:57:19.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:19.659+0000 7f544d6f4700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54480f5410 0x7f5448106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:49128/0 (socket says 172.21.15.194:49128) 2024-01-18T06:57:19.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:19.659+0000 7f544d6f4700 1 -- 172.21.15.194:0/1562832983 learned_addr learned my addr 172.21.15.194:0/1562832983 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:57:19.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:19.659+0000 7f544d6f4700 1 -- 172.21.15.194:0/1562832983 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5438005040 con 0x7f54480f5410 2024-01-18T06:57:19.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:19.660+0000 7f544d6f4700 1 --2- 172.21.15.194:0/1562832983 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54480f5410 0x7f5448106700 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f5438013510 tx=0x7f543800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:57:19.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:19.660+0000 7f543dffb700 1 -- 172.21.15.194:0/1562832983 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5438009320 con 0x7f54480f5410 2024-01-18T06:57:19.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:19.660+0000 7f544f958700 1 -- 172.21.15.194:0/1562832983 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5448106df0 con 0x7f54480f5410 2024-01-18T06:57:19.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:19.660+0000 7f543dffb700 1 -- 172.21.15.194:0/1562832983 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5438013f00 con 0x7f54480f5410 2024-01-18T06:57:19.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:19.660+0000 7f543dffb700 1 -- 172.21.15.194:0/1562832983 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5438014d90 con 0x7f54480f5410 2024-01-18T06:57:19.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:19.660+0000 7f544f958700 1 -- 172.21.15.194:0/1562832983 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5448107280 con 0x7f54480f5410 2024-01-18T06:57:19.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:19.661+0000 7f543dffb700 1 -- 172.21.15.194:0/1562832983 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f54380095f0 con 0x7f54480f5410 2024-01-18T06:57:19.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:19.661+0000 7f544f958700 1 -- 172.21.15.194:0/1562832983 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5448045ad0 con 0x7f54480f5410 2024-01-18T06:57:19.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:19.661+0000 7f543dffb700 1 --2- 172.21.15.194:0/1562832983 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f543402ff60 0x7f5434032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:57:19.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:19.661+0000 7f543dffb700 1 -- 172.21.15.194:0/1562832983 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5438041080 con 0x7f54480f5410 2024-01-18T06:57:19.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:19.661+0000 7f544cef3700 1 --2- 172.21.15.194:0/1562832983 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f543402ff60 0x7f5434032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:57:19.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:19.662+0000 7f544cef3700 1 --2- 172.21.15.194:0/1562832983 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f543402ff60 0x7f5434032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f54440096a0 tx=0x7f5444006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:57:19.665 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:19.666+0000 7f543dffb700 1 -- 172.21.15.194:0/1562832983 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5438014420 con 0x7f54480f5410 2024-01-18T06:57:19.871 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:19.872+0000 7f544f958700 1 -- 172.21.15.194:0/1562832983 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f54480fae60 con 0x7f54480f5410 2024-01-18T06:57:19.871 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:19.872+0000 7f543dffb700 1 -- 172.21.15.194:0/1562832983 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5438010230 con 0x7f54480f5410 2024-01-18T06:57:19.872 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:57:19.872 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:57:19.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:19.874+0000 7f544f958700 1 -- 172.21.15.194:0/1562832983 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f543402ff60 msgr2=0x7f5434032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:57:19.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:19.874+0000 7f544f958700 1 --2- 172.21.15.194:0/1562832983 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f543402ff60 0x7f5434032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f54440096a0 tx=0x7f5444006b40).stop 2024-01-18T06:57:19.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:19.874+0000 7f544f958700 1 -- 172.21.15.194:0/1562832983 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54480f5410 msgr2=0x7f5448106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:57:19.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:19.874+0000 7f544f958700 1 --2- 172.21.15.194:0/1562832983 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54480f5410 0x7f5448106700 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f5438013510 tx=0x7f543800b040).stop 2024-01-18T06:57:19.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:19.874+0000 7f544f958700 1 -- 172.21.15.194:0/1562832983 shutdown_connections 2024-01-18T06:57:19.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:19.874+0000 7f544f958700 1 --2- 172.21.15.194:0/1562832983 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f543402ff60 0x7f5434032420 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:57:19.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:19.874+0000 7f544f958700 1 --2- 172.21.15.194:0/1562832983 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f54480f5410 0x7f5448106700 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:57:19.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:19.874+0000 7f544f958700 1 -- 172.21.15.194:0/1562832983 >> 172.21.15.194:0/1562832983 conn(0x7f54480f0b70 msgr2=0x7f54480f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:57:19.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:19.874+0000 7f544f958700 1 -- 172.21.15.194:0/1562832983 shutdown_connections 2024-01-18T06:57:19.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:19.874+0000 7f544f958700 1 -- 172.21.15.194:0/1562832983 wait complete. 2024-01-18T06:57:19.874 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:57:20.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:57:19 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:57:19.873232+0000 mon.smithi047 (mon.0) 320 : audit [DBG] from='client.? 172.21.15.194:0/1562832983' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:57:21.257 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:57:21.258 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:57:21.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:57:20 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:57:20.059872+0000 mgr.smithi047.fuhzzr (mgr.14162) 273 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:57:22.335 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:22.334+0000 7fe6580f8700 1 -- 172.21.15.194:0/2820626374 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6500f6510 msgr2=0x7fe6500f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:57:22.335 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:22.334+0000 7fe6580f8700 1 --2- 172.21.15.194:0/2820626374 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6500f6510 0x7fe6500f6930 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7fe640009fb0 tx=0x7fe640009b30).stop 2024-01-18T06:57:22.335 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:22.334+0000 7fe6580f8700 1 -- 172.21.15.194:0/2820626374 shutdown_connections 2024-01-18T06:57:22.335 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:22.334+0000 7fe6580f8700 1 --2- 172.21.15.194:0/2820626374 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6500f6510 0x7fe6500f6930 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:57:22.335 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:22.334+0000 7fe6580f8700 1 -- 172.21.15.194:0/2820626374 >> 172.21.15.194:0/2820626374 conn(0x7fe6500f1c70 msgr2=0x7fe6500f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:57:22.335 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:22.334+0000 7fe6580f8700 1 -- 172.21.15.194:0/2820626374 shutdown_connections 2024-01-18T06:57:22.335 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:22.334+0000 7fe6580f8700 1 -- 172.21.15.194:0/2820626374 wait complete. 2024-01-18T06:57:22.335 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:22.335+0000 7fe6580f8700 1 Processor -- start 2024-01-18T06:57:22.335 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:22.335+0000 7fe6580f8700 1 -- start start 2024-01-18T06:57:22.336 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:22.335+0000 7fe6580f8700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6500f6510 0x7fe650106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:57:22.336 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:22.335+0000 7fe6580f8700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe650106c20 con 0x7fe6500f6510 2024-01-18T06:57:22.336 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:22.335+0000 7fe655e94700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6500f6510 0x7fe650106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:57:22.336 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:22.335+0000 7fe655e94700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6500f6510 0x7fe650106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:49148/0 (socket says 172.21.15.194:49148) 2024-01-18T06:57:22.336 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:22.335+0000 7fe655e94700 1 -- 172.21.15.194:0/2550017435 learned_addr learned my addr 172.21.15.194:0/2550017435 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:57:22.336 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:22.336+0000 7fe655e94700 1 -- 172.21.15.194:0/2550017435 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe640005040 con 0x7fe6500f6510 2024-01-18T06:57:22.336 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:22.336+0000 7fe655e94700 1 --2- 172.21.15.194:0/2550017435 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6500f6510 0x7fe650106730 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fe640013510 tx=0x7fe64000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:57:22.337 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:22.336+0000 7fe6467fc700 1 -- 172.21.15.194:0/2550017435 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe640009320 con 0x7fe6500f6510 2024-01-18T06:57:22.337 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:22.336+0000 7fe6580f8700 1 -- 172.21.15.194:0/2550017435 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe650106e20 con 0x7fe6500f6510 2024-01-18T06:57:22.337 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:22.336+0000 7fe6467fc700 1 -- 172.21.15.194:0/2550017435 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe640013f00 con 0x7fe6500f6510 2024-01-18T06:57:22.338 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:22.336+0000 7fe6467fc700 1 -- 172.21.15.194:0/2550017435 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe640014d90 con 0x7fe6500f6510 2024-01-18T06:57:22.338 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:22.336+0000 7fe6580f8700 1 -- 172.21.15.194:0/2550017435 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6501072b0 con 0x7fe6500f6510 2024-01-18T06:57:22.338 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:22.337+0000 7fe6467fc700 1 -- 172.21.15.194:0/2550017435 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe6400263a0 con 0x7fe6500f6510 2024-01-18T06:57:22.338 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:22.337+0000 7fe6467fc700 1 --2- 172.21.15.194:0/2550017435 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fe63c02ff60 0x7fe63c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:57:22.338 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:22.337+0000 7fe6467fc700 1 -- 172.21.15.194:0/2550017435 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe640041380 con 0x7fe6500f6510 2024-01-18T06:57:22.338 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:22.337+0000 7fe6580f8700 1 -- 172.21.15.194:0/2550017435 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe650045ad0 con 0x7fe6500f6510 2024-01-18T06:57:22.338 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:22.337+0000 7fe655693700 1 --2- 172.21.15.194:0/2550017435 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fe63c02ff60 0x7fe63c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:57:22.338 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:22.338+0000 7fe655693700 1 --2- 172.21.15.194:0/2550017435 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fe63c02ff60 0x7fe63c032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fe64c0096a0 tx=0x7fe64c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:57:22.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:22.342+0000 7fe6467fc700 1 -- 172.21.15.194:0/2550017435 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe640012920 con 0x7fe6500f6510 2024-01-18T06:57:22.546 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:22.546+0000 7fe6580f8700 1 -- 172.21.15.194:0/2550017435 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe6500fc050 con 0x7fe6500f6510 2024-01-18T06:57:22.547 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:22.547+0000 7fe6467fc700 1 -- 172.21.15.194:0/2550017435 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe64002b030 con 0x7fe6500f6510 2024-01-18T06:57:22.547 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:57:22.547 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:57:22.548 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:22.549+0000 7fe6580f8700 1 -- 172.21.15.194:0/2550017435 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fe63c02ff60 msgr2=0x7fe63c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:57:22.551 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:22.549+0000 7fe6580f8700 1 --2- 172.21.15.194:0/2550017435 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fe63c02ff60 0x7fe63c032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fe64c0096a0 tx=0x7fe64c006b40).stop 2024-01-18T06:57:22.551 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:22.549+0000 7fe6580f8700 1 -- 172.21.15.194:0/2550017435 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6500f6510 msgr2=0x7fe650106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:57:22.551 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:22.549+0000 7fe6580f8700 1 --2- 172.21.15.194:0/2550017435 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6500f6510 0x7fe650106730 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fe640013510 tx=0x7fe64000b040).stop 2024-01-18T06:57:22.551 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:22.549+0000 7fe6580f8700 1 -- 172.21.15.194:0/2550017435 shutdown_connections 2024-01-18T06:57:22.551 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:22.549+0000 7fe6580f8700 1 --2- 172.21.15.194:0/2550017435 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fe63c02ff60 0x7fe63c032420 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:57:22.552 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:22.549+0000 7fe6580f8700 1 --2- 172.21.15.194:0/2550017435 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6500f6510 0x7fe650106730 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:57:22.552 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:22.549+0000 7fe6580f8700 1 -- 172.21.15.194:0/2550017435 >> 172.21.15.194:0/2550017435 conn(0x7fe6500f1c70 msgr2=0x7fe6500f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:57:22.552 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:22.549+0000 7fe6580f8700 1 -- 172.21.15.194:0/2550017435 shutdown_connections 2024-01-18T06:57:22.552 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:22.549+0000 7fe6580f8700 1 -- 172.21.15.194:0/2550017435 wait complete. 2024-01-18T06:57:22.552 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:57:23.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:57:23 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:57:22.060125+0000 mgr.smithi047.fuhzzr (mgr.14162) 274 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:57:23.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:57:23 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:57:22.548345+0000 mon.smithi047 (mon.0) 321 : audit [DBG] from='client.? 172.21.15.194:0/2550017435' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:57:23.932 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:57:23.932 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:57:25.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:25.094+0000 7fdc56174700 1 -- 172.21.15.194:0/4231374563 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc500f5410 msgr2=0x7fdc500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:57:25.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:25.094+0000 7fdc56174700 1 --2- 172.21.15.194:0/4231374563 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc500f5410 0x7fdc500f5830 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7fdc40009fb0 tx=0x7fdc40009b30).stop 2024-01-18T06:57:25.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:25.094+0000 7fdc56174700 1 -- 172.21.15.194:0/4231374563 shutdown_connections 2024-01-18T06:57:25.094 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:25.094+0000 7fdc56174700 1 --2- 172.21.15.194:0/4231374563 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc500f5410 0x7fdc500f5830 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:57:25.094 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:25.094+0000 7fdc56174700 1 -- 172.21.15.194:0/4231374563 >> 172.21.15.194:0/4231374563 conn(0x7fdc500f0b70 msgr2=0x7fdc500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:57:25.094 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:25.094+0000 7fdc56174700 1 -- 172.21.15.194:0/4231374563 shutdown_connections 2024-01-18T06:57:25.094 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:25.094+0000 7fdc56174700 1 -- 172.21.15.194:0/4231374563 wait complete. 2024-01-18T06:57:25.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:25.094+0000 7fdc56174700 1 Processor -- start 2024-01-18T06:57:25.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:25.095+0000 7fdc56174700 1 -- start start 2024-01-18T06:57:25.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:25.095+0000 7fdc56174700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc500f5410 0x7fdc50106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:57:25.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:25.095+0000 7fdc56174700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc50106c20 con 0x7fdc500f5410 2024-01-18T06:57:25.096 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:25.095+0000 7fdc4f7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc500f5410 0x7fdc50106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:57:25.096 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:25.095+0000 7fdc4f7fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc500f5410 0x7fdc50106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:49164/0 (socket says 172.21.15.194:49164) 2024-01-18T06:57:25.096 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:25.095+0000 7fdc4f7fe700 1 -- 172.21.15.194:0/1980205475 learned_addr learned my addr 172.21.15.194:0/1980205475 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:57:25.096 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:25.095+0000 7fdc4f7fe700 1 -- 172.21.15.194:0/1980205475 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc40005040 con 0x7fdc500f5410 2024-01-18T06:57:25.096 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:25.095+0000 7fdc4f7fe700 1 --2- 172.21.15.194:0/1980205475 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc500f5410 0x7fdc50106730 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7fdc40013510 tx=0x7fdc4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:57:25.096 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:25.096+0000 7fdc37fff700 1 -- 172.21.15.194:0/1980205475 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc40009320 con 0x7fdc500f5410 2024-01-18T06:57:25.096 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:25.096+0000 7fdc56174700 1 -- 172.21.15.194:0/1980205475 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc50106e20 con 0x7fdc500f5410 2024-01-18T06:57:25.096 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:25.096+0000 7fdc37fff700 1 -- 172.21.15.194:0/1980205475 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdc40013f00 con 0x7fdc500f5410 2024-01-18T06:57:25.096 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:25.096+0000 7fdc37fff700 1 -- 172.21.15.194:0/1980205475 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc40014d90 con 0x7fdc500f5410 2024-01-18T06:57:25.096 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:25.096+0000 7fdc56174700 1 -- 172.21.15.194:0/1980205475 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc501072b0 con 0x7fdc500f5410 2024-01-18T06:57:25.097 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:25.097+0000 7fdc37fff700 1 -- 172.21.15.194:0/1980205475 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdc400095f0 con 0x7fdc500f5410 2024-01-18T06:57:25.097 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:25.097+0000 7fdc56174700 1 -- 172.21.15.194:0/1980205475 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc50045ad0 con 0x7fdc500f5410 2024-01-18T06:57:25.097 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:25.097+0000 7fdc37fff700 1 --2- 172.21.15.194:0/1980205475 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fdc3802ff60 0x7fdc38032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:57:25.097 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:25.097+0000 7fdc37fff700 1 -- 172.21.15.194:0/1980205475 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdc400410c0 con 0x7fdc500f5410 2024-01-18T06:57:25.097 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:25.097+0000 7fdc4effd700 1 --2- 172.21.15.194:0/1980205475 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fdc3802ff60 0x7fdc38032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:57:25.097 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:25.097+0000 7fdc4effd700 1 --2- 172.21.15.194:0/1980205475 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fdc3802ff60 0x7fdc38032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fdc440096a0 tx=0x7fdc44006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:57:25.101 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:25.102+0000 7fdc37fff700 1 -- 172.21.15.194:0/1980205475 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdc40014420 con 0x7fdc500f5410 2024-01-18T06:57:25.312 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:25.312+0000 7fdc56174700 1 -- 172.21.15.194:0/1980205475 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdc500fae60 con 0x7fdc500f5410 2024-01-18T06:57:25.312 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:25.313+0000 7fdc37fff700 1 -- 172.21.15.194:0/1980205475 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdc40010230 con 0x7fdc500f5410 2024-01-18T06:57:25.312 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:57:25.312 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:57:25.313 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:25.314+0000 7fdc56174700 1 -- 172.21.15.194:0/1980205475 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fdc3802ff60 msgr2=0x7fdc38032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:57:25.314 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:25.314+0000 7fdc56174700 1 --2- 172.21.15.194:0/1980205475 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fdc3802ff60 0x7fdc38032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fdc440096a0 tx=0x7fdc44006b40).stop 2024-01-18T06:57:25.314 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:25.314+0000 7fdc56174700 1 -- 172.21.15.194:0/1980205475 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc500f5410 msgr2=0x7fdc50106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:57:25.314 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:25.314+0000 7fdc56174700 1 --2- 172.21.15.194:0/1980205475 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc500f5410 0x7fdc50106730 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7fdc40013510 tx=0x7fdc4000b040).stop 2024-01-18T06:57:25.314 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:25.314+0000 7fdc56174700 1 -- 172.21.15.194:0/1980205475 shutdown_connections 2024-01-18T06:57:25.314 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:25.314+0000 7fdc56174700 1 --2- 172.21.15.194:0/1980205475 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7fdc3802ff60 0x7fdc38032420 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:57:25.314 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:25.314+0000 7fdc56174700 1 --2- 172.21.15.194:0/1980205475 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc500f5410 0x7fdc50106730 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:57:25.314 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:25.314+0000 7fdc56174700 1 -- 172.21.15.194:0/1980205475 >> 172.21.15.194:0/1980205475 conn(0x7fdc500f0b70 msgr2=0x7fdc500f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:57:25.314 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:25.314+0000 7fdc56174700 1 -- 172.21.15.194:0/1980205475 shutdown_connections 2024-01-18T06:57:25.314 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:25.314+0000 7fdc56174700 1 -- 172.21.15.194:0/1980205475 wait complete. 2024-01-18T06:57:25.315 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:57:25.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:57:25 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:57:24.060478+0000 mgr.smithi047.fuhzzr (mgr.14162) 275 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:57:26.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:57:26 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:57:25.313695+0000 mon.smithi047 (mon.0) 322 : audit [DBG] from='client.? 172.21.15.194:0/1980205475' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:57:26.702 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:57:26.702 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:57:27.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:57:27 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:57:26.060889+0000 mgr.smithi047.fuhzzr (mgr.14162) 276 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:57:27.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:27.873+0000 7f72da07a700 1 -- 172.21.15.194:0/4001484738 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72d40f6510 msgr2=0x7f72d40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:57:27.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:27.873+0000 7f72da07a700 1 --2- 172.21.15.194:0/4001484738 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72d40f6510 0x7f72d40f6930 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f72c8009fb0 tx=0x7f72c8009b30).stop 2024-01-18T06:57:27.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:27.873+0000 7f72da07a700 1 -- 172.21.15.194:0/4001484738 shutdown_connections 2024-01-18T06:57:27.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:27.873+0000 7f72da07a700 1 --2- 172.21.15.194:0/4001484738 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72d40f6510 0x7f72d40f6930 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:57:27.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:27.873+0000 7f72da07a700 1 -- 172.21.15.194:0/4001484738 >> 172.21.15.194:0/4001484738 conn(0x7f72d40f1c70 msgr2=0x7f72d40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:57:27.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:27.873+0000 7f72da07a700 1 -- 172.21.15.194:0/4001484738 shutdown_connections 2024-01-18T06:57:27.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:27.873+0000 7f72da07a700 1 -- 172.21.15.194:0/4001484738 wait complete. 2024-01-18T06:57:27.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:27.874+0000 7f72da07a700 1 Processor -- start 2024-01-18T06:57:27.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:27.874+0000 7f72da07a700 1 -- start start 2024-01-18T06:57:27.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:27.874+0000 7f72da07a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72d40f6510 0x7f72d4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:57:27.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:27.874+0000 7f72da07a700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72d4106c20 con 0x7f72d40f6510 2024-01-18T06:57:27.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:27.874+0000 7f72d37fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72d40f6510 0x7f72d4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:57:27.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:27.875+0000 7f72d37fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72d40f6510 0x7f72d4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:49204/0 (socket says 172.21.15.194:49204) 2024-01-18T06:57:27.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:27.875+0000 7f72d37fe700 1 -- 172.21.15.194:0/2965499442 learned_addr learned my addr 172.21.15.194:0/2965499442 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:57:27.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:27.875+0000 7f72d37fe700 1 -- 172.21.15.194:0/2965499442 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72c8005040 con 0x7f72d40f6510 2024-01-18T06:57:27.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:27.875+0000 7f72d37fe700 1 --2- 172.21.15.194:0/2965499442 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72d40f6510 0x7f72d4106730 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f72c8013510 tx=0x7f72c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:57:27.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:27.875+0000 7f72bbfff700 1 -- 172.21.15.194:0/2965499442 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72c8009320 con 0x7f72d40f6510 2024-01-18T06:57:27.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:27.875+0000 7f72da07a700 1 -- 172.21.15.194:0/2965499442 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72d4106e20 con 0x7f72d40f6510 2024-01-18T06:57:27.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:27.875+0000 7f72bbfff700 1 -- 172.21.15.194:0/2965499442 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f72c8013f00 con 0x7f72d40f6510 2024-01-18T06:57:27.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:27.875+0000 7f72bbfff700 1 -- 172.21.15.194:0/2965499442 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72c8014d90 con 0x7f72d40f6510 2024-01-18T06:57:27.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:27.875+0000 7f72da07a700 1 -- 172.21.15.194:0/2965499442 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72d41072b0 con 0x7f72d40f6510 2024-01-18T06:57:27.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:27.876+0000 7f72bbfff700 1 -- 172.21.15.194:0/2965499442 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f72c80095f0 con 0x7f72d40f6510 2024-01-18T06:57:27.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:27.876+0000 7f72da07a700 1 -- 172.21.15.194:0/2965499442 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72d4045ad0 con 0x7f72d40f6510 2024-01-18T06:57:27.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:27.876+0000 7f72bbfff700 1 --2- 172.21.15.194:0/2965499442 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f72bc02ff60 0x7f72bc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:57:27.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:27.876+0000 7f72bbfff700 1 -- 172.21.15.194:0/2965499442 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f72c80410c0 con 0x7f72d40f6510 2024-01-18T06:57:27.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:27.877+0000 7f72d2ffd700 1 --2- 172.21.15.194:0/2965499442 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f72bc02ff60 0x7f72bc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:57:27.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:27.877+0000 7f72d2ffd700 1 --2- 172.21.15.194:0/2965499442 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f72bc02ff60 0x7f72bc032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f72c40096a0 tx=0x7f72c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:57:27.880 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:27.881+0000 7f72bbfff700 1 -- 172.21.15.194:0/2965499442 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f72c8014420 con 0x7f72d40f6510 2024-01-18T06:57:28.086 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:28.086+0000 7f72da07a700 1 -- 172.21.15.194:0/2965499442 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f72d40fbf60 con 0x7f72d40f6510 2024-01-18T06:57:28.086 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:28.087+0000 7f72bbfff700 1 -- 172.21.15.194:0/2965499442 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f72c801d7f0 con 0x7f72d40f6510 2024-01-18T06:57:28.087 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:57:28.087 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:57:28.088 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:28.089+0000 7f72da07a700 1 -- 172.21.15.194:0/2965499442 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f72bc02ff60 msgr2=0x7f72bc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:57:28.088 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:28.089+0000 7f72da07a700 1 --2- 172.21.15.194:0/2965499442 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f72bc02ff60 0x7f72bc032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f72c40096a0 tx=0x7f72c4006b40).stop 2024-01-18T06:57:28.088 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:28.089+0000 7f72da07a700 1 -- 172.21.15.194:0/2965499442 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72d40f6510 msgr2=0x7f72d4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:57:28.088 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:28.089+0000 7f72da07a700 1 --2- 172.21.15.194:0/2965499442 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72d40f6510 0x7f72d4106730 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f72c8013510 tx=0x7f72c800b040).stop 2024-01-18T06:57:28.089 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:28.089+0000 7f72da07a700 1 -- 172.21.15.194:0/2965499442 shutdown_connections 2024-01-18T06:57:28.089 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:28.089+0000 7f72da07a700 1 --2- 172.21.15.194:0/2965499442 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f72bc02ff60 0x7f72bc032420 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:57:28.089 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:28.089+0000 7f72da07a700 1 --2- 172.21.15.194:0/2965499442 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72d40f6510 0x7f72d4106730 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:57:28.089 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:28.089+0000 7f72da07a700 1 -- 172.21.15.194:0/2965499442 >> 172.21.15.194:0/2965499442 conn(0x7f72d40f1c70 msgr2=0x7f72d40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:57:28.089 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:28.089+0000 7f72da07a700 1 -- 172.21.15.194:0/2965499442 shutdown_connections 2024-01-18T06:57:28.089 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:28.089+0000 7f72da07a700 1 -- 172.21.15.194:0/2965499442 wait complete. 2024-01-18T06:57:28.089 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:57:29.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:57:28 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:57:28.061100+0000 mgr.smithi047.fuhzzr (mgr.14162) 277 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:57:29.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:57:28 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:57:28.088280+0000 mon.smithi047 (mon.0) 323 : audit [DBG] from='client.? 172.21.15.194:0/2965499442' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:57:29.472 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:57:29.473 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:57:30.626 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:30.626+0000 7f5e150ff700 1 -- 172.21.15.194:0/410957097 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e100f51a0 msgr2=0x7f5e100f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:57:30.626 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:30.626+0000 7f5e150ff700 1 --2- 172.21.15.194:0/410957097 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e100f51a0 0x7f5e100f55c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f5e00009fb0 tx=0x7f5e00009b30).stop 2024-01-18T06:57:30.626 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:30.626+0000 7f5e150ff700 1 -- 172.21.15.194:0/410957097 shutdown_connections 2024-01-18T06:57:30.626 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:30.626+0000 7f5e150ff700 1 --2- 172.21.15.194:0/410957097 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e100f51a0 0x7f5e100f55c0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:57:30.626 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:30.626+0000 7f5e150ff700 1 -- 172.21.15.194:0/410957097 >> 172.21.15.194:0/410957097 conn(0x7f5e100f09a0 msgr2=0x7f5e100f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:57:30.627 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:30.626+0000 7f5e150ff700 1 -- 172.21.15.194:0/410957097 shutdown_connections 2024-01-18T06:57:30.627 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:30.626+0000 7f5e150ff700 1 -- 172.21.15.194:0/410957097 wait complete. 2024-01-18T06:57:30.627 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:30.626+0000 7f5e150ff700 1 Processor -- start 2024-01-18T06:57:30.627 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:30.627+0000 7f5e150ff700 1 -- start start 2024-01-18T06:57:30.627 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:30.627+0000 7f5e150ff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e100f51a0 0x7f5e101086b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:57:30.627 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:30.627+0000 7f5e150ff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e10108ba0 con 0x7f5e100f51a0 2024-01-18T06:57:30.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:30.627+0000 7f5e0ffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e100f51a0 0x7f5e101086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:57:30.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:30.627+0000 7f5e0ffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e100f51a0 0x7f5e101086b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:49222/0 (socket says 172.21.15.194:49222) 2024-01-18T06:57:30.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:30.627+0000 7f5e0ffff700 1 -- 172.21.15.194:0/233100787 learned_addr learned my addr 172.21.15.194:0/233100787 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:57:30.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:30.627+0000 7f5e0ffff700 1 -- 172.21.15.194:0/233100787 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e00005040 con 0x7f5e100f51a0 2024-01-18T06:57:30.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:30.628+0000 7f5e0ffff700 1 --2- 172.21.15.194:0/233100787 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e100f51a0 0x7f5e101086b0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f5e00013510 tx=0x7f5e0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:57:30.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:30.628+0000 7f5e0cff9700 1 -- 172.21.15.194:0/233100787 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e00009320 con 0x7f5e100f51a0 2024-01-18T06:57:30.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:30.628+0000 7f5e150ff700 1 -- 172.21.15.194:0/233100787 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e10108da0 con 0x7f5e100f51a0 2024-01-18T06:57:30.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:30.628+0000 7f5e0cff9700 1 -- 172.21.15.194:0/233100787 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e00013f00 con 0x7f5e100f51a0 2024-01-18T06:57:30.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:30.628+0000 7f5e150ff700 1 -- 172.21.15.194:0/233100787 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e10109230 con 0x7f5e100f51a0 2024-01-18T06:57:30.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:30.628+0000 7f5e0cff9700 1 -- 172.21.15.194:0/233100787 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e00014dc0 con 0x7f5e100f51a0 2024-01-18T06:57:30.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:30.629+0000 7f5e150ff700 1 -- 172.21.15.194:0/233100787 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e10045ad0 con 0x7f5e100f51a0 2024-01-18T06:57:30.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:30.629+0000 7f5e0cff9700 1 -- 172.21.15.194:0/233100787 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5e000095f0 con 0x7f5e100f51a0 2024-01-18T06:57:30.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:30.629+0000 7f5e0cff9700 1 --2- 172.21.15.194:0/233100787 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f5df802ff60 0x7f5df8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:57:30.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:30.629+0000 7f5e0cff9700 1 -- 172.21.15.194:0/233100787 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5e00040ff0 con 0x7f5e100f51a0 2024-01-18T06:57:30.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:30.629+0000 7f5e0f7fe700 1 --2- 172.21.15.194:0/233100787 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f5df802ff60 0x7f5df8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:57:30.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:30.630+0000 7f5e0f7fe700 1 --2- 172.21.15.194:0/233100787 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f5df802ff60 0x7f5df8032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f5e040096a0 tx=0x7f5e04006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:57:30.633 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:30.634+0000 7f5e0cff9700 1 -- 172.21.15.194:0/233100787 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5e00014420 con 0x7f5e100f51a0 2024-01-18T06:57:30.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:30.840+0000 7f5e150ff700 1 -- 172.21.15.194:0/233100787 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5e1010a080 con 0x7f5e100f51a0 2024-01-18T06:57:30.840 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:30.841+0000 7f5e0cff9700 1 -- 172.21.15.194:0/233100787 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5e00010230 con 0x7f5e100f51a0 2024-01-18T06:57:30.841 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:57:30.841 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:57:30.842 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:30.842+0000 7f5e150ff700 1 -- 172.21.15.194:0/233100787 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f5df802ff60 msgr2=0x7f5df8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:57:30.842 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:30.842+0000 7f5e150ff700 1 --2- 172.21.15.194:0/233100787 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f5df802ff60 0x7f5df8032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f5e040096a0 tx=0x7f5e04006b40).stop 2024-01-18T06:57:30.842 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:30.842+0000 7f5e150ff700 1 -- 172.21.15.194:0/233100787 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e100f51a0 msgr2=0x7f5e101086b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:57:30.842 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:30.842+0000 7f5e150ff700 1 --2- 172.21.15.194:0/233100787 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e100f51a0 0x7f5e101086b0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f5e00013510 tx=0x7f5e0000b040).stop 2024-01-18T06:57:30.842 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:30.842+0000 7f5e150ff700 1 -- 172.21.15.194:0/233100787 shutdown_connections 2024-01-18T06:57:30.842 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:30.842+0000 7f5e150ff700 1 --2- 172.21.15.194:0/233100787 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f5df802ff60 0x7f5df8032420 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:57:30.842 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:30.842+0000 7f5e150ff700 1 --2- 172.21.15.194:0/233100787 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e100f51a0 0x7f5e101086b0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:57:30.842 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:30.842+0000 7f5e150ff700 1 -- 172.21.15.194:0/233100787 >> 172.21.15.194:0/233100787 conn(0x7f5e100f09a0 msgr2=0x7f5e100f1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:57:30.842 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:30.842+0000 7f5e150ff700 1 -- 172.21.15.194:0/233100787 shutdown_connections 2024-01-18T06:57:30.842 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:30.842+0000 7f5e150ff700 1 -- 172.21.15.194:0/233100787 wait complete. 2024-01-18T06:57:30.843 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:57:31.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:57:31 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:57:30.061446+0000 mgr.smithi047.fuhzzr (mgr.14162) 278 : cluster [DBG] pgmap v264: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:57:31.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:57:31 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:57:30.121046+0000 mon.smithi047 (mon.0) 324 : audit [INF] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.fuhzzr/mirror_snapshot_schedule"}]: dispatch 2024-01-18T06:57:31.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:57:31 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:57:30.121276+0000 mon.smithi047 (mon.0) 325 : audit [INF] from='mgr.14162 172.21.15.47:0/1594286738' entity='mgr.smithi047.fuhzzr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi047.fuhzzr/trash_purge_schedule"}]: dispatch 2024-01-18T06:57:31.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:57:31 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:57:30.841747+0000 mon.smithi047 (mon.0) 326 : audit [DBG] from='client.? 172.21.15.194:0/233100787' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:57:32.224 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T06:57:32.224 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T06:57:33.344 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:33.344+0000 7f6d71e7d700 1 -- 172.21.15.194:0/1830795813 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d6c0f51a0 msgr2=0x7f6d6c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:57:33.344 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:33.344+0000 7f6d71e7d700 1 --2- 172.21.15.194:0/1830795813 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d6c0f51a0 0x7f6d6c0f55c0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f6d60009fb0 tx=0x7f6d60009b30).stop 2024-01-18T06:57:33.344 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:33.344+0000 7f6d71e7d700 1 -- 172.21.15.194:0/1830795813 shutdown_connections 2024-01-18T06:57:33.344 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:33.344+0000 7f6d71e7d700 1 --2- 172.21.15.194:0/1830795813 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d6c0f51a0 0x7f6d6c0f55c0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:57:33.344 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:33.344+0000 7f6d71e7d700 1 -- 172.21.15.194:0/1830795813 >> 172.21.15.194:0/1830795813 conn(0x7f6d6c0f09a0 msgr2=0x7f6d6c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:57:33.344 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:33.344+0000 7f6d71e7d700 1 -- 172.21.15.194:0/1830795813 shutdown_connections 2024-01-18T06:57:33.345 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:33.344+0000 7f6d71e7d700 1 -- 172.21.15.194:0/1830795813 wait complete. 2024-01-18T06:57:33.345 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:33.344+0000 7f6d71e7d700 1 Processor -- start 2024-01-18T06:57:33.345 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:33.344+0000 7f6d71e7d700 1 -- start start 2024-01-18T06:57:33.345 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:33.344+0000 7f6d71e7d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d6c0f51a0 0x7f6d6c1091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:57:33.345 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:33.344+0000 7f6d71e7d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d6c060220 con 0x7f6d6c0f51a0 2024-01-18T06:57:33.345 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:33.345+0000 7f6d70e7b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d6c0f51a0 0x7f6d6c1091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:57:33.345 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:33.345+0000 7f6d70e7b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d6c0f51a0 0x7f6d6c1091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.194:49234/0 (socket says 172.21.15.194:49234) 2024-01-18T06:57:33.345 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:33.345+0000 7f6d70e7b700 1 -- 172.21.15.194:0/3910788857 learned_addr learned my addr 172.21.15.194:0/3910788857 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-18T06:57:33.346 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:33.345+0000 7f6d70e7b700 1 -- 172.21.15.194:0/3910788857 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d60005040 con 0x7f6d6c0f51a0 2024-01-18T06:57:33.347 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:33.345+0000 7f6d70e7b700 1 --2- 172.21.15.194:0/3910788857 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d6c0f51a0 0x7f6d6c1091d0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f6d60013510 tx=0x7f6d6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:57:33.347 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:33.345+0000 7f6d697fa700 1 -- 172.21.15.194:0/3910788857 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d60009320 con 0x7f6d6c0f51a0 2024-01-18T06:57:33.347 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:33.345+0000 7f6d71e7d700 1 -- 172.21.15.194:0/3910788857 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d6c060420 con 0x7f6d6c0f51a0 2024-01-18T06:57:33.347 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:33.345+0000 7f6d697fa700 1 -- 172.21.15.194:0/3910788857 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6d60013f00 con 0x7f6d6c0f51a0 2024-01-18T06:57:33.347 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:33.345+0000 7f6d697fa700 1 -- 172.21.15.194:0/3910788857 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d60014dc0 con 0x7f6d6c0f51a0 2024-01-18T06:57:33.347 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:33.345+0000 7f6d71e7d700 1 -- 172.21.15.194:0/3910788857 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d6c109880 con 0x7f6d6c0f51a0 2024-01-18T06:57:33.347 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:33.346+0000 7f6d697fa700 1 -- 172.21.15.194:0/3910788857 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6d600095f0 con 0x7f6d6c0f51a0 2024-01-18T06:57:33.347 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:33.346+0000 7f6d71e7d700 1 -- 172.21.15.194:0/3910788857 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d6c059100 con 0x7f6d6c0f51a0 2024-01-18T06:57:33.347 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:33.346+0000 7f6d697fa700 1 --2- 172.21.15.194:0/3910788857 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f6d5c02ffb0 0x7f6d5c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T06:57:33.347 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:33.346+0000 7f6d697fa700 1 -- 172.21.15.194:0/3910788857 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6d600410b0 con 0x7f6d6c0f51a0 2024-01-18T06:57:33.347 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:33.347+0000 7f6d6bfff700 1 --2- 172.21.15.194:0/3910788857 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f6d5c02ffb0 0x7f6d5c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T06:57:33.347 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:33.347+0000 7f6d6bfff700 1 --2- 172.21.15.194:0/3910788857 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f6d5c02ffb0 0x7f6d5c032470 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f6d580096a0 tx=0x7f6d58006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T06:57:33.350 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:33.351+0000 7f6d697fa700 1 -- 172.21.15.194:0/3910788857 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6d6006d2b0 con 0x7f6d6c0f51a0 2024-01-18T06:57:33.463 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:57:33 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: cluster 2024-01-18T06:57:32.061645+0000 mgr.smithi047.fuhzzr (mgr.14162) 279 : cluster [DBG] pgmap v265: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T06:57:33.556 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:33.556+0000 7f6d71e7d700 1 -- 172.21.15.194:0/3910788857 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6d6c0fabe0 con 0x7f6d6c0f51a0 2024-01-18T06:57:33.556 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:33.557+0000 7f6d697fa700 1 -- 172.21.15.194:0/3910788857 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6d60014420 con 0x7f6d6c0f51a0 2024-01-18T06:57:33.557 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-18T06:57:33.557 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"653e4012-b5cd-11ee-95ae-87774f69a715","modified":"2024-01-18T06:47:31.951147Z","created":"2024-01-18T06:47:31.951147Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi047","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:3300","nonce":0},{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T06:57:33.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:33.559+0000 7f6d71e7d700 1 -- 172.21.15.194:0/3910788857 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f6d5c02ffb0 msgr2=0x7f6d5c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:57:33.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:33.559+0000 7f6d71e7d700 1 --2- 172.21.15.194:0/3910788857 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f6d5c02ffb0 0x7f6d5c032470 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f6d580096a0 tx=0x7f6d58006b40).stop 2024-01-18T06:57:33.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:33.559+0000 7f6d71e7d700 1 -- 172.21.15.194:0/3910788857 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d6c0f51a0 msgr2=0x7f6d6c1091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T06:57:33.559 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:33.559+0000 7f6d71e7d700 1 --2- 172.21.15.194:0/3910788857 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d6c0f51a0 0x7f6d6c1091d0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f6d60013510 tx=0x7f6d6000b040).stop 2024-01-18T06:57:33.559 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:33.559+0000 7f6d71e7d700 1 -- 172.21.15.194:0/3910788857 shutdown_connections 2024-01-18T06:57:33.559 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:33.559+0000 7f6d71e7d700 1 --2- 172.21.15.194:0/3910788857 >> [v2:172.21.15.47:6800/1933549322,v1:172.21.15.47:6801/1933549322] conn(0x7f6d5c02ffb0 0x7f6d5c032470 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:57:33.559 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:33.559+0000 7f6d71e7d700 1 --2- 172.21.15.194:0/3910788857 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d6c0f51a0 0x7f6d6c1091d0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T06:57:33.559 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:33.559+0000 7f6d71e7d700 1 -- 172.21.15.194:0/3910788857 >> 172.21.15.194:0/3910788857 conn(0x7f6d6c0f09a0 msgr2=0x7f6d6c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T06:57:33.559 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:33.559+0000 7f6d71e7d700 1 -- 172.21.15.194:0/3910788857 shutdown_connections 2024-01-18T06:57:33.559 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-18T06:57:33.559+0000 7f6d71e7d700 1 -- 172.21.15.194:0/3910788857 wait complete. 2024-01-18T06:57:33.559 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-18T06:57:33.988 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2024-01-18T06:57:33.990 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-01-18T06:57:33.990 DEBUG:teuthology.orchestra.run.smithi047:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-18T06:57:34.014 DEBUG:teuthology.orchestra.run.smithi194:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-18T06:57:34.032 INFO:tasks.cephadm:Stopping all daemons... 2024-01-18T06:57:34.032 INFO:tasks.cephadm.mon.smithi047:Stopping mon.smithi047... 2024-01-18T06:57:34.032 DEBUG:teuthology.orchestra.run.smithi047:> sudo systemctl stop ceph-653e4012-b5cd-11ee-95ae-87774f69a715@mon.smithi047 2024-01-18T06:57:34.442 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:57:34 smithi047 systemd[1]: Stopping Ceph mon.smithi047 for 653e4012-b5cd-11ee-95ae-87774f69a715... 2024-01-18T06:57:34.442 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:57:34 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: audit 2024-01-18T06:57:33.558098+0000 mon.smithi047 (mon.0) 327 : audit [DBG] from='client.? 172.21.15.194:0/3910788857' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T06:57:34.442 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:57:34 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:57:34.171+0000 7f4eb47d1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi047 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-01-18T06:57:34.442 INFO:journalctl@ceph.mon.smithi047.smithi047.stdout:Jan 18 06:57:34 smithi047 ceph-653e4012-b5cd-11ee-95ae-87774f69a715-mon.smithi047[38797]: debug 2024-01-18T06:57:34.171+0000 7f4eb47d1700 -1 mon.smithi047@0(leader) e1 *** Got Signal Terminated *** 2024-01-18T06:57:34.644 DEBUG:teuthology.orchestra.run.smithi047:> sudo pkill -f 'journalctl -f -n 0 -u ceph-653e4012-b5cd-11ee-95ae-87774f69a715@mon.smithi047.service' 2024-01-18T06:57:34.695 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-18T06:57:34.695 INFO:tasks.cephadm.mon.smithi047:Stopped mon.smithi047 2024-01-18T06:57:34.695 INFO:tasks.cephadm.mon.smithi194:Stopping mon.smithi194... 2024-01-18T06:57:34.695 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-653e4012-b5cd-11ee-95ae-87774f69a715@mon.smithi194 2024-01-18T06:57:34.731 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to stop ceph-653e4012-b5cd-11ee-95ae-87774f69a715@mon.smithi194.service: Unit ceph-653e4012-b5cd-11ee-95ae-87774f69a715@mon.smithi194.service not loaded. 2024-01-18T06:57:34.732 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-01-18T06:57:34.733 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi194" Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi194 with status 5: 'sudo systemctl stop ceph-653e4012-b5cd-11ee-95ae-87774f69a715@mon.smithi194' 2024-01-18T06:57:34.734 INFO:tasks.cephadm:Archiving crash dumps... 2024-01-18T06:57:34.736 DEBUG:teuthology.misc:Transferring archived files from smithi047:/var/lib/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-01-17_22:43:41-rados-wip-yuri10-testing-2024-01-17-0759-pacific-distro-default-smithi/7519629/remote/ubuntu@smithi047.front.sepia.ceph.com/crash 2024-01-18T06:57:34.737 DEBUG:teuthology.orchestra.run.smithi047:> sudo tar c -f - -C /var/lib/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/crash -- . 2024-01-18T06:57:34.777 DEBUG:teuthology.misc:Transferring archived files from smithi194:/var/lib/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-01-17_22:43:41-rados-wip-yuri10-testing-2024-01-17-0759-pacific-distro-default-smithi/7519629/remote/ubuntu@smithi194.front.sepia.ceph.com/crash 2024-01-18T06:57:34.778 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /var/lib/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/crash -- . 2024-01-18T06:57:34.813 INFO:teuthology.orchestra.run.smithi194.stderr:tar: /var/lib/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/crash: Cannot open: No such file or directory 2024-01-18T06:57:34.813 INFO:teuthology.orchestra.run.smithi194.stderr:tar: Error is not recoverable: exiting now 2024-01-18T06:57:34.814 INFO:tasks.cephadm:Checking cluster log for badness... 2024-01-18T06:57:34.815 DEBUG:teuthology.orchestra.run.smithi047:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-01-18T06:57:34.852 INFO:teuthology.orchestra.run.smithi047.stderr:grep: /var/log/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/ceph.log: No such file or directory 2024-01-18T06:57:34.854 INFO:tasks.cephadm:Compressing logs... 2024-01-18T06:57:34.854 DEBUG:teuthology.orchestra.run.smithi047:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-18T06:57:34.896 DEBUG:teuthology.orchestra.run.smithi194:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-18T06:57:34.923 INFO:teuthology.orchestra.run.smithi047.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-01-18T06:57:34.923 INFO:teuthology.orchestra.run.smithi047.stderr:: No such file or directory 2024-01-18T06:57:34.923 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/ceph-mon.smithi047.log 2024-01-18T06:57:34.924 INFO:teuthology.orchestra.run.smithi194.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-01-18T06:57:34.925 INFO:teuthology.orchestra.run.smithi194.stderr:: No such file or directory 2024-01-18T06:57:34.925 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/ceph-volume.log 2024-01-18T06:57:34.925 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/ceph-mgr.smithi047.fuhzzr.log 2024-01-18T06:57:34.925 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/ceph-mon.smithi047.log: /var/log/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/ceph-volume.log: /var/log/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/ceph-mgr.smithi047.fuhzzr.log: 90.2% -- replaced with /var/log/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/ceph-volume.log.gz 2024-01-18T06:57:34.926 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/cephadm.log: 96.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-01-18T06:57:34.927 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-01-18T06:57:34.927 INFO:teuthology.orchestra.run.smithi194.stderr:real 0m0.014s 2024-01-18T06:57:34.927 INFO:teuthology.orchestra.run.smithi194.stderr:user 0m0.011s 2024-01-18T06:57:34.927 INFO:teuthology.orchestra.run.smithi194.stderr:sys 0m0.011s 2024-01-18T06:57:34.929 INFO:teuthology.orchestra.run.smithi047.stderr: 92.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-01-18T06:57:35.004 INFO:teuthology.orchestra.run.smithi047.stderr: 92.6% -- replaced with /var/log/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/ceph-mgr.smithi047.fuhzzr.log.gz 2024-01-18T06:57:35.072 INFO:teuthology.orchestra.run.smithi047.stderr: 90.6% -- replaced with /var/log/ceph/653e4012-b5cd-11ee-95ae-87774f69a715/ceph-mon.smithi047.log.gz 2024-01-18T06:57:35.073 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-01-18T06:57:35.073 INFO:teuthology.orchestra.run.smithi047.stderr:real 0m0.161s 2024-01-18T06:57:35.073 INFO:teuthology.orchestra.run.smithi047.stderr:user 0m0.234s 2024-01-18T06:57:35.074 INFO:teuthology.orchestra.run.smithi047.stderr:sys 0m0.023s 2024-01-18T06:57:35.074 INFO:tasks.cephadm:Archiving logs... 2024-01-18T06:57:35.074 DEBUG:teuthology.misc:Transferring archived files from smithi047:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-17_22:43:41-rados-wip-yuri10-testing-2024-01-17-0759-pacific-distro-default-smithi/7519629/remote/ubuntu@smithi047.front.sepia.ceph.com/log 2024-01-18T06:57:35.076 DEBUG:teuthology.orchestra.run.smithi047:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-18T06:57:35.249 DEBUG:teuthology.misc:Transferring archived files from smithi194:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-17_22:43:41-rados-wip-yuri10-testing-2024-01-17-0759-pacific-distro-default-smithi/7519629/remote/ubuntu@smithi194.front.sepia.ceph.com/log 2024-01-18T06:57:35.251 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-18T06:57:35.290 INFO:tasks.cephadm:Removing cluster... 2024-01-18T06:57:35.290 DEBUG:teuthology.orchestra.run.smithi047:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 --force 2024-01-18T06:58:25.696 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 653e4012-b5cd-11ee-95ae-87774f69a715 --force 2024-01-18T06:58:25.996 INFO:tasks.cephadm:Removing cephadm ... 2024-01-18T06:58:25.997 DEBUG:teuthology.orchestra.run.smithi047:> rm -rf /home/ubuntu/cephtest/cephadm 2024-01-18T06:58:26.017 DEBUG:teuthology.orchestra.run.smithi194:> rm -rf /home/ubuntu/cephtest/cephadm 2024-01-18T06:58:26.035 INFO:tasks.cephadm:Teardown complete 2024-01-18T06:58:26.035 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi194 with status 5: 'sudo systemctl stop ceph-653e4012-b5cd-11ee-95ae-87774f69a715@mon.smithi194' 2024-01-18T06:58:26.361 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=4bc560a290384f94900f826ffafe66b5 Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi194 with status 5: 'sudo systemctl stop ceph-653e4012-b5cd-11ee-95ae-87774f69a715@mon.smithi194' 2024-01-18T06:58:26.366 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-01-18T06:58:26.375 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-18T06:58:26.393 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-18T06:58:26.393 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-18T06:58:26.420 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-18T06:58:26.502 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-18T06:58:26.512 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-18T06:58:26.512 DEBUG:teuthology.orchestra.run.smithi047:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-18T06:58:26.515 DEBUG:teuthology.orchestra.run.smithi194:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-18T06:58:26.532 INFO:teuthology.orchestra.run.smithi194.stderr:bash: ntpq: command not found 2024-01-18T06:58:26.533 INFO:teuthology.orchestra.run.smithi047.stderr:bash: ntpq: command not found 2024-01-18T06:58:26.536 INFO:teuthology.orchestra.run.smithi194.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-18T06:58:26.536 INFO:teuthology.orchestra.run.smithi194.stdout:=============================================================================== 2024-01-18T06:58:26.536 INFO:teuthology.orchestra.run.smithi194.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 39 -299us[ -299us] +/- 84ms 2024-01-18T06:58:26.536 INFO:teuthology.orchestra.run.smithi194.stdout:^* hv02.front.sepia.ceph.com 3 6 377 41 -1754us[-1541us] +/- 72ms 2024-01-18T06:58:26.537 INFO:teuthology.orchestra.run.smithi194.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 40 +1929us[+1929us] +/- 66ms 2024-01-18T06:58:26.537 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-18T06:58:26.538 INFO:teuthology.orchestra.run.smithi047.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-18T06:58:26.538 INFO:teuthology.orchestra.run.smithi047.stdout:=============================================================================== 2024-01-18T06:58:26.539 INFO:teuthology.orchestra.run.smithi047.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 40 -289us[ -265us] +/- 84ms 2024-01-18T06:58:26.539 INFO:teuthology.orchestra.run.smithi047.stdout:^* hv02.front.sepia.ceph.com 3 6 377 38 -1739us[-1714us] +/- 72ms 2024-01-18T06:58:26.539 INFO:teuthology.orchestra.run.smithi047.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 41 +1957us[+1981us] +/- 66ms 2024-01-18T06:58:26.539 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-18T06:58:26.539 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-18T06:58:26.549 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-18T06:58:26.549 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-18T06:58:26.558 DEBUG:teuthology.orchestra.run.smithi047:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-18T06:58:26.646 DEBUG:teuthology.orchestra.run.smithi194:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-18T06:58:26.709 DEBUG:teuthology.orchestra.run.smithi047:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-18T06:58:26.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-18T06:58:26.741 DEBUG:teuthology.orchestra.run.smithi194:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-18T06:58:26.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-18T06:58:26.771 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-18T06:58:26.781 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-18T06:58:26.790 INFO:teuthology.task.internal:Duration was 1230.210620 seconds 2024-01-18T06:58:26.790 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-18T06:58:26.799 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-18T06:58:26.799 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-18T06:58:26.802 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-18T06:58:26.849 INFO:teuthology.orchestra.run.smithi047.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-18T06:58:26.857 INFO:teuthology.orchestra.run.smithi194.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-18T06:58:27.226 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-18T06:58:27.226 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi047.front.sepia.ceph.com 2024-01-18T06:58:27.226 DEBUG:teuthology.orchestra.run.smithi047:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-18T06:58:27.251 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi194.front.sepia.ceph.com 2024-01-18T06:58:27.251 DEBUG:teuthology.orchestra.run.smithi194:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-18T06:58:27.274 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-18T06:58:27.275 DEBUG:teuthology.orchestra.run.smithi047:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-18T06:58:27.294 DEBUG:teuthology.orchestra.run.smithi194:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-18T06:58:27.363 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-18T06:58:27.363 DEBUG:teuthology.orchestra.run.smithi047:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-18T06:58:27.365 DEBUG:teuthology.orchestra.run.smithi194:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-18T06:58:27.453 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-18T06:58:27.464 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-18T06:58:27.464 DEBUG:teuthology.orchestra.run.smithi047:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-18T06:58:27.492 DEBUG:teuthology.orchestra.run.smithi194:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-18T06:58:27.520 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-18T06:58:27.540 DEBUG:teuthology.orchestra.run.smithi047:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-18T06:58:27.542 DEBUG:teuthology.orchestra.run.smithi194:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-18T06:58:27.567 INFO:teuthology.orchestra.run.smithi047.stdout:kernel.core_pattern = core 2024-01-18T06:58:27.587 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern = core 2024-01-18T06:58:27.621 DEBUG:teuthology.orchestra.run.smithi047:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-18T06:58:27.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-18T06:58:27.655 DEBUG:teuthology.orchestra.run.smithi194:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-18T06:58:27.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-18T06:58:27.678 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-18T06:58:27.689 INFO:teuthology.task.internal:Transferring archived files... 2024-01-18T06:58:27.690 DEBUG:teuthology.misc:Transferring archived files from smithi047:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-17_22:43:41-rados-wip-yuri10-testing-2024-01-17-0759-pacific-distro-default-smithi/7519629/remote/smithi047 2024-01-18T06:58:27.691 DEBUG:teuthology.orchestra.run.smithi047:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-18T06:58:27.735 DEBUG:teuthology.misc:Transferring archived files from smithi194:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-17_22:43:41-rados-wip-yuri10-testing-2024-01-17-0759-pacific-distro-default-smithi/7519629/remote/smithi194 2024-01-18T06:58:27.736 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-18T06:58:27.775 INFO:teuthology.task.internal:Removing archive directory... 2024-01-18T06:58:27.776 DEBUG:teuthology.orchestra.run.smithi047:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-18T06:58:27.778 DEBUG:teuthology.orchestra.run.smithi194:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-18T06:58:27.827 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-18T06:58:27.839 INFO:teuthology.task.internal:Not uploading archives. 2024-01-18T06:58:27.839 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-18T06:58:27.851 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-18T06:58:27.851 DEBUG:teuthology.orchestra.run.smithi047:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-18T06:58:27.854 DEBUG:teuthology.orchestra.run.smithi194:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-18T06:58:27.869 INFO:teuthology.orchestra.run.smithi047.stdout: 525353 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 18 06:58 /home/ubuntu/cephtest 2024-01-18T06:58:27.885 INFO:teuthology.orchestra.run.smithi194.stdout: 525353 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 18 06:58 /home/ubuntu/cephtest 2024-01-18T06:58:27.886 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-18T06:58:27.896 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-18T06:58:27.916 INFO:teuthology.nuke:Checking targets against current locks 2024-01-18T06:58:27.956 DEBUG:teuthology.nuke:shortname: smithi047 2024-01-18T06:58:27.956 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-18T06:58:27.979 DEBUG:teuthology.nuke:shortname: smithi194 2024-01-18T06:58:27.979 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-18T06:58:27.983 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi047.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-17_22:43:41-rados-wip-yuri10-testing-2024-01-17-0759-pacific-distro-default-smithi/7519629', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-18 06:29:59.808262', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDwqTlovo+UQwwudhGXWkdlbOj3jbu+mFjgFuy+Gum24E77mpPYOVbPuFmy3dpfKHHDBKtVCTwbC5ZH+n8ZXucI='} 2024-01-18T06:58:28.008 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi194.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-17_22:43:41-rados-wip-yuri10-testing-2024-01-17-0759-pacific-distro-default-smithi/7519629', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-18 06:29:59.806539', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO6tf9GNGxt7c16eAsvnjeb9dAGmQe1i8V7sc2WV7fuPyEFm1sh0M3FFdYCRyQvEp0ntsTICFFtBzvDczjuRGD4='} 2024-01-18T06:58:28.040 INFO:teuthology.orchestra.console.smithi047:Power off 2024-01-18T06:58:28.041 DEBUG:teuthology.orchestra.console.smithi047:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-18T06:58:28.056 INFO:teuthology.orchestra.console.smithi194:Power off 2024-01-18T06:58:28.057 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-18T06:58:28.066 DEBUG:teuthology.orchestra.console.smithi047:power off output: Chassis Power Control: Down/Off 2024-01-18T06:58:28.067 DEBUG:teuthology.orchestra.console.smithi047:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-18T06:58:28.079 DEBUG:teuthology.orchestra.console.smithi194:power off output: Chassis Power Control: Down/Off 2024-01-18T06:58:28.080 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-18T06:58:28.092 DEBUG:teuthology.orchestra.console.smithi047:check power output: Chassis Power is on 2024-01-18T06:58:28.103 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is on 2024-01-18T06:58:32.093 DEBUG:teuthology.orchestra.console.smithi047:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-18T06:58:32.104 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-18T06:58:32.203 DEBUG:teuthology.orchestra.console.smithi047:check power output: Chassis Power is on 2024-01-18T06:58:32.214 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is on 2024-01-18T06:58:36.208 DEBUG:teuthology.orchestra.console.smithi047:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-18T06:58:36.216 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-18T06:58:36.319 DEBUG:teuthology.orchestra.console.smithi047:check power output: Chassis Power is off 2024-01-18T06:58:36.326 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is off 2024-01-18T06:58:36.420 INFO:teuthology.orchestra.console.smithi047:Power off completed 2024-01-18T06:58:36.427 INFO:teuthology.orchestra.console.smithi194:Power off completed 2024-01-18T06:58:36.531 INFO:teuthology.run:Summary data: description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1230.2106199264526 failure_reason: 'Command failed on smithi194 with status 5: ''sudo systemctl stop ceph-653e4012-b5cd-11ee-95ae-87774f69a715@mon.smithi194''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=4bc560a290384f94900f826ffafe66b5 status: fail success: false 2024-01-18T06:58:36.532 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-18T06:58:36.623 INFO:teuthology.run:FAIL